ConcurrentRebalancersTest.TwoConcurrentRebalancers/1: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 4
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240501 08:17:48.494513  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10280
I20240501 08:17:48.501084 10399 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:48.513931 10207 catalog_manager.cc:4573] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3453d8d883014219b1b14c88e250736b: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:17:48.609162 10619 raft_consensus.cc:1203] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20240501 08:17:48.687883 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:48.698424 25439 ts_tablet_manager.cc:902] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:17:48.700023 25439 tablet_copy_client.cc:308] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:17:48.700639 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 21065b7dcc18456eb1b4c2376d51eec4 from peer 861f629e2d134ff08f25fa553dfb5bda ({username='slave'} at 127.2.248.130:34215)
I20240501 08:17:48.700789 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 21065b7dcc18456eb1b4c2376d51eec4 from peer 861f629e2d134ff08f25fa553dfb5bda at {username='slave'} at 127.2.248.130:34215: session id = 861f629e2d134ff08f25fa553dfb5bda-21065b7dcc18456eb1b4c2376d51eec4
I20240501 08:17:48.703307 10772 tablet_copy_source_session.cc:205] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:48.704546 25439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21065b7dcc18456eb1b4c2376d51eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:48.709208 25439 tablet_copy_client.cc:783] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:48.714463 25439 tablet_copy_client.cc:647] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:48.716030 25439 tablet_copy_client.cc:523] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:48.718361 25439 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:17:48.735456 25439 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:48.735596 25050 consensus_queue.cc:579] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.412s)
I20240501 08:17:48.735895 25439 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap complete.
I20240501 08:17:48.736202 25439 ts_tablet_manager.cc:1367] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.018s	user 0.016s	sys 0.003s
I20240501 08:17:48.736969 25439 raft_consensus.cc:348] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:48.737251 25439 raft_consensus.cc:729] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: LEARNER
I20240501 08:17:48.737461 25439 consensus_queue.cc:260] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:48.737977 25439 ts_tablet_manager.cc:1398] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:17:48.739334 10488 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:17:48.741385 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session 861f629e2d134ff08f25fa553dfb5bda-21065b7dcc18456eb1b4c2376d51eec4 received from {username='slave'} at 127.2.248.130:34215
I20240501 08:17:48.741485 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session 861f629e2d134ff08f25fa553dfb5bda-21065b7dcc18456eb1b4c2376d51eec4 on tablet 21065b7dcc18456eb1b4c2376d51eec4 with peer 861f629e2d134ff08f25fa553dfb5bda
I20240501 08:17:48.741442 10619 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:17:48.742429 25432 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:17:48.744966 24218 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } }
I20240501 08:17:48.747570 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d4ab8e42a484cf89b42f5f53f511265 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:48.748025 10619 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } }
I20240501 08:17:48.750038 10218 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 2 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:48.756409 10488 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:48.759609 10425 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.759653 10619 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:17:48.760416 24218 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:17:48.762265 25050 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.762774 10619 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.765810 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6d4ab8e42a484cf89b42f5f53f511265 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:48.767361 10218 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 2 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.851586 25426 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b6a964e54f3b44eda6d0666e0c1e4bea)
I20240501 08:17:48.851771 25426 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:48.852450 25426 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
W20240501 08:17:48.853878 10685 leader_election.cc:334] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:48.868825 10358 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:17:48.870982 10362 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:17:48.872606 24218 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:17:48.873643 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:17:48.874177 10687 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06c1439c9ab8415485eff20566811ede; no voters: b6a964e54f3b44eda6d0666e0c1e4bea, c3747ec2a93c4684962f215616e896ed
I20240501 08:17:48.874527 25423 raft_consensus.cc:2726] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:17:48.875355 10358 raft_consensus.cc:858] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:48.875900 10358 raft_consensus.cc:1565] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Could not prepare op 'term: 2 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:48.876729 10295 consensus_peers.cc:462] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:48.886068 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4): 4921 bytes on disk
I20240501 08:17:48.886888 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240501 08:17:48.896705 10358 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } }
W20240501 08:17:48.902441 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:48.904560 10358 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.937062 25139 raft_consensus.cc:1013] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer 06c1439c9ab8415485eff20566811ede: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:17:48.943101 25453 ts_tablet_manager.cc:902] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
W20240501 08:17:48.949239 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:48.963023 25453 tablet_copy_client.cc:308] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:17:48.964155 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf ({username='slave'} at 127.2.248.131:44669)
I20240501 08:17:48.964358 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf at {username='slave'} at 127.2.248.131:44669: session id = 8d5f781e3ba94cbfbbea9b5782ef8eaf-6e56f5f3cd9f4a56a3d43543ac0e5d14
I20240501 08:17:48.967959 10381 tablet_copy_source_session.cc:205] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:48.969630 25453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:48.978621 25453 tablet_copy_client.cc:783] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:48.984762 25453 tablet_copy_client.cc:647] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:48.985970 25432 raft_consensus.cc:1050] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER 06c1439c9ab8415485eff20566811ede to VOTER
I20240501 08:17:48.986827 25453 tablet_copy_client.cc:523] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:48.987269 25432 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } }
I20240501 08:17:48.989881 25453 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:17:48.990180 10749 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:48.990811 25050 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:17:48.992224 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.993350 10619 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:48.994503 25050 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:17:48.998302 25050 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } } }
I20240501 08:17:48.999248 10619 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } } }
I20240501 08:17:49.007637 10749 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } } }
I20240501 08:17:49.009409 10218 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:49.010708 25426 consensus_queue.cc:579] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.084s)
I20240501 08:17:49.018540 10749 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
W20240501 08:17:49.030900 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:49.033902 10358 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:49.034960 25428 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:17:49.039378 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:49.040902 25428 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:49.045074 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:49.045766 10358 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:49.051034 10218 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 9 to 10, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:49.054756 25453 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:49.055151 25453 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap complete.
I20240501 08:17:49.055433 25453 ts_tablet_manager.cc:1367] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.066s	user 0.038s	sys 0.019s
I20240501 08:17:49.056181 25453 raft_consensus.cc:348] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
I20240501 08:17:49.056895 25453 raft_consensus.cc:729] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: LEARNER
I20240501 08:17:49.057463 25453 consensus_queue.cc:260] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
I20240501 08:17:49.058844 25453 ts_tablet_manager.cc:1398] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
W20240501 08:17:49.058497 10295 consensus_peers.cc:462] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.060070 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session 8d5f781e3ba94cbfbbea9b5782ef8eaf-6e56f5f3cd9f4a56a3d43543ac0e5d14 received from {username='slave'} at 127.2.248.131:44669
I20240501 08:17:49.060232 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session 8d5f781e3ba94cbfbbea9b5782ef8eaf-6e56f5f3cd9f4a56a3d43543ac0e5d14 on tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 with peer 8d5f781e3ba94cbfbbea9b5782ef8eaf
I20240501 08:17:49.104575 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): 4581 bytes on disk
I20240501 08:17:49.105216 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 08:17:49.130558 10492 raft_consensus.cc:1203] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240501 08:17:49.140324 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
W20240501 08:17:49.157265 10557 consensus_peers.cc:485] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.201092 25050 raft_consensus.cc:1050] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf to VOTER
I20240501 08:17:49.201757 25050 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } }
I20240501 08:17:49.204108 10749 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:17:49.205085 10619 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:17:49.205075 25050 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240501 08:17:49.206233 25050 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:17:49.206502 10425 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:49.210173 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.210642 24218 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.211548 10619 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.214277 10749 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.219579 25447 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b6a964e54f3b44eda6d0666e0c1e4bea)
I20240501 08:17:49.219843 25447 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:49.221019 25447 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
I20240501 08:17:49.223078 10218 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.224290 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:17:49.224568 10749 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 2.
W20240501 08:17:49.224606 10295 leader_election.cc:334] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.226182 10297 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:17:49.228401 25447 raft_consensus.cc:2781] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:17:49.228519 25447 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader b6a964e54f3b44eda6d0666e0c1e4bea)
I20240501 08:17:49.228653 25447 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:17:49.231871 25447 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:49.233088 25447 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
W20240501 08:17:49.234838 10295 leader_election.cc:334] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.234009 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:17:49.235810 10749 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:17:49.240084 10749 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 3.
I20240501 08:17:49.240626 10297 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:17:49.240967 25447 raft_consensus.cc:2781] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:17:49.241154 25447 raft_consensus.cc:686] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 3 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:17:49.241535 25447 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:49.250631 10221 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 2 to 3, leader changed from b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 3 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:49.326258 10619 raft_consensus.cc:1203] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240501 08:17:49.357326 25137 raft_consensus.cc:1013] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer 06c1439c9ab8415485eff20566811ede: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 08:17:49.377364 10295 consensus_peers.cc:462] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.399592 10210 maintenance_manager.cc:419] P ee59fa3365af4b91907d4ab87092780e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10424 bytes on disk
I20240501 08:17:49.403685 10209 maintenance_manager.cc:643] P ee59fa3365af4b91907d4ab87092780e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s	user 0.004s	sys 0.000s Metrics: {"thread_start_us":231,"threads_started":1}
W20240501 08:17:49.482102 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:49.497063  3042 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10280: Timed out: CheckLeaks RPC to 127.2.248.129:44049 timed out after 1.000s (SENT)
I20240501 08:17:49.497287  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10280
W20240501 08:17:49.516992 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:49.518471 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.557607  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10410
W20240501 08:17:49.562269 10207 catalog_manager.cc:4573] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3453d8d883014219b1b14c88e250736b: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:17:49.565677 10532 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:49.649559 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.681677 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.706872 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.749640 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
W20240501 08:17:49.793797 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.828708 25137 raft_consensus.cc:1013] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer 06c1439c9ab8415485eff20566811ede: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 08:17:49.894850 10687 connection.cc:535] client connection to 127.2.248.129:44049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:49.896448 10687 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:49.896579 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:49.933763 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.936336 10557 consensus_peers.cc:485] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:49.941272  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10410
W20240501 08:17:49.954964 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:50.002514 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
LLVM ERROR: IO failure on output stream: Broken pipe
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcdfce)[0x55b53b8e6fce]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc27d)[0x55b53b8e527d]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc3d3)[0x55b53b8e53d3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f057d628980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f057c706fb7]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f057c708921]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa575c)[0x55b53b8be75c]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa58e0)[0x55b53b8be8e0]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xbcc73)[0x55b53b8d5c73]
/lib/x86_64-linux-gnu/libc.so.6(+0x43161)[0x7f057c70b161]
/lib/x86_64-linux-gnu/libc.so.6(+0x4325a)[0x7f057c70b25a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xee)[0x7f057c6e9bfe]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0x8f0ba)[0x55b53b8a80ba]
W20240501 08:17:50.021636 10687 connection.cc:535] server connection from 127.2.248.130:34215 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:50.021683 10685 connection.cc:535] client connection to 127.2.248.130:46605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:50.021867 10555 connection.cc:535] client connection to 127.2.248.130:46605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:17:50.022436  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10540
W20240501 08:17:50.023203 10555 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.023204 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.023329 10685 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.023334 10555 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:50.029034 10662 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:50.033936 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.139729 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:50.178439 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
W20240501 08:17:50.256088 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:50.405244 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:17:50.406117  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10540
W20240501 08:17:50.406148 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:50.457290 10685 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:50.461288 10685 connection.cc:535] client connection to 127.2.248.131:34319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:17:50.461697  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10670
W20240501 08:17:50.462452 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): Couldn't send request to peer 8d5f781e3ba94cbfbbea9b5782ef8eaf. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:50.467391 10789 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:50.508368 10205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34319: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 08:17:50.564241 10205 catalog_manager.cc:4573] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): ChangeConfig:ADD_PEER:NON_VOTER RPC failed for tablet 03d0f89e1d8f40fe84d275b117d8145f: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34319: connect: Connection refused (error 111)
I20240501 08:17:50.793329  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10670
I20240501 08:17:50.846485  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10190
I20240501 08:17:50.851821 10249 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:17:51.103031  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10190
2024-05-01T08:17:51Z chronyd exiting
I20240501 08:17:51.169014  3042 test_util.cc:175] -----------------------------------------------
I20240501 08:17:51.169097  3042 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0

Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN      ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:10:38.336953  3042 test_util.cc:268] Using random seed: 93773709
I20240501 08:10:38.348345  3042 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:38.348430  3042 ts_itest-base.cc:113] --------------
I20240501 08:10:38.348476  3042 ts_itest-base.cc:114] 6 tablet servers
I20240501 08:10:38.348543  3042 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:10:38.348618  3042 ts_itest-base.cc:116] --------------
2024-05-01T08:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:38Z Disabled control of system clock
I20240501 08:10:38.372640  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:42765
--webserver_interface=127.2.248.190
--webserver_port=0
--builtin_ntp_servers=127.2.248.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.248.190:42765
--location_mapping_cmd=/tmp/dist-test-task3vHYYh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:10:38.690830  3054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:38.691133  3054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:38.691277  3054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:38.698592  3054 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:38.698719  3054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:38.698809  3054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:38.698889  3054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:38.698953  3054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:38.707934  3054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task3vHYYh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.2.248.190:42765
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.248.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:38.709228  3054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:38.710925  3054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:38.722353  3059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:38.722833  3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:38.723564  3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:38.724236  3054 server_base.cc:1034] running on GCE node
I20240501 08:10:38.724970  3054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:38.726436  3054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:38.727631  3054 hybrid_clock.cc:648] HybridClock initialized: now 1714551038727593 us; error 32 us; skew 500 ppm
I20240501 08:10:38.727952  3054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:38.734405  3054 webserver.cc:460] Webserver started at http://127.2.248.190:33581/ using document root <none> and password file <none>
I20240501 08:10:38.735162  3054 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:38.735272  3054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:38.735582  3054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:38.738229  3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3079d9ef376b44d28c2fee88d70d0d5a"
format_stamp: "Formatted at 2024-05-01 08:10:38 on dist-test-slave-0pdg"
I20240501 08:10:38.738950  3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3079d9ef376b44d28c2fee88d70d0d5a"
format_stamp: "Formatted at 2024-05-01 08:10:38 on dist-test-slave-0pdg"
I20240501 08:10:38.743921  3054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240501 08:10:38.747138  3068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:38.748467  3054 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:10:38.748605  3054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3079d9ef376b44d28c2fee88d70d0d5a"
format_stamp: "Formatted at 2024-05-01 08:10:38 on dist-test-slave-0pdg"
I20240501 08:10:38.748777  3054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:38.787240  3054 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:10:38.788105  3054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:38.788425  3054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:38.806291  3054 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.190:42765
I20240501 08:10:38.806310  3119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.190:42765 every 8 connection(s)
I20240501 08:10:38.807646  3054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:38.808740  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 3054
I20240501 08:10:38.808990  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:38.811604  3120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:38.822218  3120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a: Bootstrap starting.
I20240501 08:10:38.825277  3120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:38.826251  3120 log.cc:824] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:38.828750  3120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a: No bootstrap required, opened a new log
I20240501 08:10:38.835743  3120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 42765 } }
I20240501 08:10:38.836074  3120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:38.836237  3120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3079d9ef376b44d28c2fee88d70d0d5a, State: Initialized, Role: FOLLOWER
I20240501 08:10:38.836813  3120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 42765 } }
I20240501 08:10:38.837009  3120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:38.837132  3120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:38.837265  3120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:38.839231  3120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 42765 } }
I20240501 08:10:38.839670  3120 leader_election.cc:302] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3079d9ef376b44d28c2fee88d70d0d5a; no voters: 
I20240501 08:10:38.840361  3120 leader_election.cc:288] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:10:38.840394  3125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:38.840994  3125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [term 1 LEADER]: Becoming Leader. State: Replica: 3079d9ef376b44d28c2fee88d70d0d5a, State: Running, Role: LEADER
I20240501 08:10:38.841634  3125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 42765 } }
I20240501 08:10:38.842852  3120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:38.844468  3127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3079d9ef376b44d28c2fee88d70d0d5a. Latest consensus state: current_term: 1 leader_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 42765 } } }
I20240501 08:10:38.844513  3126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3079d9ef376b44d28c2fee88d70d0d5a" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 42765 } } }
I20240501 08:10:38.844811  3127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:38.844993  3126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:38.848762  3130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:38.854673  3130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:38.863272  3130 catalog_manager.cc:1347] Generated new cluster ID: 8ff555345b90472ab13a0cf1735d046e
I20240501 08:10:38.863394  3130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:38.878968  3130 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:38.880165  3130 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:38.901844  3130 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3079d9ef376b44d28c2fee88d70d0d5a: Generated new TSK 0
I20240501 08:10:38.902798  3130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:38.918253  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.129:0
--local_ip_for_outbound_sockets=127.2.248.129
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--builtin_ntp_servers=127.2.248.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:10:39.260393  3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:39.260713  3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:39.260846  3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:39.268523  3144 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:39.268669  3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:39.268831  3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.129
I20240501 08:10:39.277595  3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.129
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:39.278925  3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:39.280645  3144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:39.292217  3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.292773  3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.293396  3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.294049  3144 server_base.cc:1034] running on GCE node
I20240501 08:10:39.294605  3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:39.295518  3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:39.296706  3144 hybrid_clock.cc:648] HybridClock initialized: now 1714551039296670 us; error 39 us; skew 500 ppm
I20240501 08:10:39.296988  3144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:39.299096  3144 webserver.cc:460] Webserver started at http://127.2.248.129:39225/ using document root <none> and password file <none>
I20240501 08:10:39.299800  3144 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:39.299908  3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:39.300252  3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:39.302690  3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-0pdg"
I20240501 08:10:39.303411  3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-0pdg"
I20240501 08:10:39.308398  3144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240501 08:10:39.311573  3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:39.312841  3144 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:10:39.312987  3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-0pdg"
I20240501 08:10:39.313167  3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:39.358212  3144 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:10:39.359083  3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:39.359354  3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:39.360323  3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:39.361807  3144 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:39.361902  3144 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:39.362015  3144 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:39.362084  3144 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:39.392750  3144 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.129:32925
I20240501 08:10:39.392800  3270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.129:32925 every 8 connection(s)
I20240501 08:10:39.393973  3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:39.397064  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 3144
I20240501 08:10:39.397275  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:39.401254  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.130:0
--local_ip_for_outbound_sockets=127.2.248.130
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--builtin_ntp_servers=127.2.248.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:39.407653  3271 heartbeater.cc:347] Connected to a master server at 127.2.248.190:42765
I20240501 08:10:39.407907  3271 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:39.408730  3271 heartbeater.cc:511] Master 127.2.248.190:42765 requested a full tablet report, sending...
I20240501 08:10:39.452101  3085 ts_manager.cc:194] Registered new tserver with Master: c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:39.454612  3085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.129:47205
W20240501 08:10:39.719388  3275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:39.719672  3275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:39.719758  3275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:39.726853  3275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:39.726971  3275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:39.727110  3275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.130
I20240501 08:10:39.735832  3275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.130
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:39.737120  3275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:39.738767  3275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:39.750488  3282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.751159  3284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.751750  3281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.752499  3275 server_base.cc:1034] running on GCE node
I20240501 08:10:39.753085  3275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:39.753942  3275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:39.755152  3275 hybrid_clock.cc:648] HybridClock initialized: now 1714551039755110 us; error 39 us; skew 500 ppm
I20240501 08:10:39.755456  3275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:39.757555  3275 webserver.cc:460] Webserver started at http://127.2.248.130:35143/ using document root <none> and password file <none>
I20240501 08:10:39.758246  3275 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:39.758363  3275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:39.758658  3275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:39.761334  3275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1979c6dfa70f42699a57e8210a28b878"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-0pdg"
I20240501 08:10:39.762336  3275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1979c6dfa70f42699a57e8210a28b878"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-0pdg"
I20240501 08:10:39.769042  3275 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20240501 08:10:39.772388  3290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:39.773623  3275 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240501 08:10:39.773744  3275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1979c6dfa70f42699a57e8210a28b878"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-0pdg"
I20240501 08:10:39.773901  3275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:39.813023  3275 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:10:39.814049  3275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:39.814332  3275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:39.815486  3275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:39.817286  3275 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:39.817394  3275 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:39.817510  3275 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:39.817589  3275 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:39.848048  3275 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.130:33957
I20240501 08:10:39.848179  3402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.130:33957 every 8 connection(s)
I20240501 08:10:39.849344  3275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:39.858654  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 3275
I20240501 08:10:39.858804  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:39.861892  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:0
--local_ip_for_outbound_sockets=127.2.248.131
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--builtin_ntp_servers=127.2.248.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:39.862174  3403 heartbeater.cc:347] Connected to a master server at 127.2.248.190:42765
I20240501 08:10:39.862486  3403 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:39.863569  3403 heartbeater.cc:511] Master 127.2.248.190:42765 requested a full tablet report, sending...
I20240501 08:10:39.904551  3085 ts_manager.cc:194] Registered new tserver with Master: 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:39.905936  3085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.130:43465
W20240501 08:10:40.175657  3406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:40.175942  3406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:40.176081  3406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:40.184785  3406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:40.184913  3406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:40.185058  3406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:10:40.193650  3406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:40.194864  3406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:40.196578  3406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:40.207963  3412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.208515  3415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.209045  3413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:40.209717  3406 server_base.cc:1034] running on GCE node
I20240501 08:10:40.210208  3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:40.211017  3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:40.212236  3406 hybrid_clock.cc:648] HybridClock initialized: now 1714551040212170 us; error 52 us; skew 500 ppm
I20240501 08:10:40.212568  3406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:40.214787  3406 webserver.cc:460] Webserver started at http://127.2.248.131:35659/ using document root <none> and password file <none>
I20240501 08:10:40.215538  3406 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:40.215639  3406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:40.215973  3406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:40.218664  3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "97336c9b436644a798689a443b2f272e"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-0pdg"
I20240501 08:10:40.219427  3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "97336c9b436644a798689a443b2f272e"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-0pdg"
I20240501 08:10:40.224557  3406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20240501 08:10:40.227656  3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:40.229146  3406 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:10:40.229322  3406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "97336c9b436644a798689a443b2f272e"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-0pdg"
I20240501 08:10:40.229533  3406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:40.248773  3406 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:10:40.249572  3406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:40.249809  3406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:40.250691  3406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:40.252100  3406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:40.252221  3406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:40.252339  3406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:40.252420  3406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:40.282827  3406 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:44211
I20240501 08:10:40.282878  3533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:44211 every 8 connection(s)
I20240501 08:10:40.284044  3406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:40.289933  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 3406
I20240501 08:10:40.290140  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:40.293097  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:0
--local_ip_for_outbound_sockets=127.2.248.132
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--builtin_ntp_servers=127.2.248.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:40.296195  3534 heartbeater.cc:347] Connected to a master server at 127.2.248.190:42765
I20240501 08:10:40.296456  3534 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:40.297430  3534 heartbeater.cc:511] Master 127.2.248.190:42765 requested a full tablet report, sending...
I20240501 08:10:40.336903  3085 ts_manager.cc:194] Registered new tserver with Master: 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:40.337836  3085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:42491
I20240501 08:10:40.457890  3271 heartbeater.cc:503] Master 127.2.248.190:42765 was elected leader, sending a full tablet report...
W20240501 08:10:40.602159  3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:40.602481  3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:40.602631  3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:40.609898  3537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:40.610034  3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:40.610198  3537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:10:40.618880  3537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:40.620220  3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:40.621871  3537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:40.633846  3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.634341  3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.635001  3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:40.635684  3537 server_base.cc:1034] running on GCE node
I20240501 08:10:40.636301  3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:40.637219  3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:40.638417  3537 hybrid_clock.cc:648] HybridClock initialized: now 1714551040638368 us; error 46 us; skew 500 ppm
I20240501 08:10:40.638710  3537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:40.640945  3537 webserver.cc:460] Webserver started at http://127.2.248.132:36651/ using document root <none> and password file <none>
I20240501 08:10:40.641638  3537 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:40.641746  3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:40.642074  3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:40.644838  3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "29eea43c04ae49edb64ba8803bf09b14"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-0pdg"
I20240501 08:10:40.645686  3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "29eea43c04ae49edb64ba8803bf09b14"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-0pdg"
I20240501 08:10:40.650817  3537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240501 08:10:40.654330  3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:40.655611  3537 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:40.655757  3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "29eea43c04ae49edb64ba8803bf09b14"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-0pdg"
I20240501 08:10:40.655941  3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:40.688447  3537 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:10:40.689296  3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:40.689561  3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:40.690479  3537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:40.691920  3537 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:40.692016  3537 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:40.692159  3537 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:40.692242  3537 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:40.722513  3537 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:37267
I20240501 08:10:40.722572  3664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:37267 every 8 connection(s)
I20240501 08:10:40.723750  3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:40.729583  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 3537
I20240501 08:10:40.729753  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:40.732652  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:0
--local_ip_for_outbound_sockets=127.2.248.133
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--builtin_ntp_servers=127.2.248.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:40.736966  3665 heartbeater.cc:347] Connected to a master server at 127.2.248.190:42765
I20240501 08:10:40.737309  3665 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:40.738307  3665 heartbeater.cc:511] Master 127.2.248.190:42765 requested a full tablet report, sending...
I20240501 08:10:40.778231  3085 ts_manager.cc:194] Registered new tserver with Master: 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:40.779512  3085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:41685
I20240501 08:10:40.908807  3403 heartbeater.cc:503] Master 127.2.248.190:42765 was elected leader, sending a full tablet report...
W20240501 08:10:41.045362  3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.045637  3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.045725  3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.052927  3668 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:41.053068  3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.053251  3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:10:41.062033  3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:41.063282  3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:41.064882  3668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.076439  3675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.077035  3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.077634  3674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.078286  3668 server_base.cc:1034] running on GCE node
I20240501 08:10:41.078933  3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:41.079903  3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:41.081096  3668 hybrid_clock.cc:648] HybridClock initialized: now 1714551041081026 us; error 64 us; skew 500 ppm
I20240501 08:10:41.081401  3668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.083490  3668 webserver.cc:460] Webserver started at http://127.2.248.133:37749/ using document root <none> and password file <none>
I20240501 08:10:41.084185  3668 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.084292  3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.084574  3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.086968  3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b526e10277944ab0b4744f9de18a8771"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-0pdg"
I20240501 08:10:41.087687  3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b526e10277944ab0b4744f9de18a8771"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-0pdg"
I20240501 08:10:41.092787  3668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240501 08:10:41.095953  3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:41.097220  3668 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:10:41.097371  3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b526e10277944ab0b4744f9de18a8771"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-0pdg"
I20240501 08:10:41.097550  3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.116451  3668 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:10:41.117298  3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.117547  3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.118582  3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.120066  3668 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.120177  3668 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20240501 08:10:41.120278  3668 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.120342  3668 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:41.150266  3668 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:37767
I20240501 08:10:41.150332  3795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:37767 every 8 connection(s)
I20240501 08:10:41.151590  3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:41.159015  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 3668
I20240501 08:10:41.159209  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:41.162686  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.134:0
--local_ip_for_outbound_sockets=127.2.248.134
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--builtin_ntp_servers=127.2.248.148:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:41.164635  3796 heartbeater.cc:347] Connected to a master server at 127.2.248.190:42765
I20240501 08:10:41.165002  3796 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:41.166066  3796 heartbeater.cc:511] Master 127.2.248.190:42765 requested a full tablet report, sending...
I20240501 08:10:41.208951  3085 ts_manager.cc:194] Registered new tserver with Master: b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:10:41.209918  3085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:45151
I20240501 08:10:41.340745  3534 heartbeater.cc:503] Master 127.2.248.190:42765 was elected leader, sending a full tablet report...
W20240501 08:10:41.490607  3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.490896  3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.491037  3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.498193  3799 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:41.498327  3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.498492  3799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.134
I20240501 08:10:41.507126  3799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:42765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.134
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:41.508373  3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:41.509974  3799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.521364  3805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.521847  3808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.522604  3806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.523373  3799 server_base.cc:1034] running on GCE node
I20240501 08:10:41.523948  3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:41.524837  3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:41.526031  3799 hybrid_clock.cc:648] HybridClock initialized: now 1714551041525988 us; error 42 us; skew 500 ppm
I20240501 08:10:41.526314  3799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.528487  3799 webserver.cc:460] Webserver started at http://127.2.248.134:44171/ using document root <none> and password file <none>
I20240501 08:10:41.529171  3799 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.529279  3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.529573  3799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.532064  3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6deb22ef32e2476cb56efabd79a948c0"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-0pdg"
I20240501 08:10:41.533454  3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6deb22ef32e2476cb56efabd79a948c0"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-0pdg"
I20240501 08:10:41.538403  3799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:10:41.541744  3814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:41.543201  3799 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:10:41.543357  3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6deb22ef32e2476cb56efabd79a948c0"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-0pdg"
I20240501 08:10:41.543557  3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.575492  3799 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:10:41.576334  3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.576586  3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.577541  3799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.578998  3799 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.579082  3799 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:41.579171  3799 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.579222  3799 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:10:41.609707  3799 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.134:37605
I20240501 08:10:41.609793  3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.134:37605 every 8 connection(s)
I20240501 08:10:41.610981  3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:10:41.620762  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 3799
I20240501 08:10:41.620967  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:10:41.623746  3927 heartbeater.cc:347] Connected to a master server at 127.2.248.190:42765
I20240501 08:10:41.624058  3927 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:41.625015  3927 heartbeater.cc:511] Master 127.2.248.190:42765 requested a full tablet report, sending...
I20240501 08:10:41.667538  3085 ts_manager.cc:194] Registered new tserver with Master: 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:10:41.668469  3085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.134:45443
I20240501 08:10:41.675729  3042 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 08:10:41.696605  3042 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 08:10:41.782122  3665 heartbeater.cc:503] Master 127.2.248.190:42765 was elected leader, sending a full tablet report...
I20240501 08:10:42.213114  3796 heartbeater.cc:503] Master 127.2.248.190:42765 was elected leader, sending a full tablet report...
I20240501 08:10:42.671108  3927 heartbeater.cc:503] Master 127.2.248.190:42765 was elected leader, sending a full tablet report...
I20240501 08:10:45.450321  3085 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:45.495299  3469 tablet_service.cc:1467] Processing CreateTablet for tablet d66bace0bf96429eab97117c872963e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.496856  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66bace0bf96429eab97117c872963e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.497984  3862 tablet_service.cc:1467] Processing CreateTablet for tablet d66bace0bf96429eab97117c872963e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.498615  3468 tablet_service.cc:1467] Processing CreateTablet for tablet b6d31a9a1fe048dda14ce04e0050644e (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.499183  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d31a9a1fe048dda14ce04e0050644e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.501601  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66bace0bf96429eab97117c872963e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.510382  3205 tablet_service.cc:1467] Processing CreateTablet for tablet b6d31a9a1fe048dda14ce04e0050644e (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.511103  3731 tablet_service.cc:1467] Processing CreateTablet for tablet b3a51a508c09466ebc399b8441cd3f64 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.512521  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a51a508c09466ebc399b8441cd3f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.512986  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d31a9a1fe048dda14ce04e0050644e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.517144  3730 tablet_service.cc:1467] Processing CreateTablet for tablet b6d31a9a1fe048dda14ce04e0050644e (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.517715  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d31a9a1fe048dda14ce04e0050644e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.520502  3206 tablet_service.cc:1467] Processing CreateTablet for tablet d66bace0bf96429eab97117c872963e1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.521090  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66bace0bf96429eab97117c872963e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.519641  3338 tablet_service.cc:1467] Processing CreateTablet for tablet b3a51a508c09466ebc399b8441cd3f64 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.522468  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a51a508c09466ebc399b8441cd3f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.532332  3600 tablet_service.cc:1467] Processing CreateTablet for tablet b3a51a508c09466ebc399b8441cd3f64 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47946d2a755d4e65af22d6f03e3b171a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:45.540015  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a51a508c09466ebc399b8441cd3f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.547101  3951 tablet_bootstrap.cc:492] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:45.549952  3955 tablet_bootstrap.cc:492] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:45.552506  3952 tablet_bootstrap.cc:492] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:45.553510  3955 tablet_bootstrap.cc:654] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.553499  3951 tablet_bootstrap.cc:654] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.554733  3955 log.cc:824] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:45.554975  3951 log.cc:824] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:45.559340  3952 tablet_bootstrap.cc:654] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.567135  3952 log.cc:824] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:45.568625  3954 tablet_bootstrap.cc:492] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:45.575044  3951 tablet_bootstrap.cc:492] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:45.575484  3951 ts_tablet_manager.cc:1367] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.004s
I20240501 08:10:45.576411  3954 tablet_bootstrap.cc:654] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.578902  3952 tablet_bootstrap.cc:492] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:45.579293  3952 ts_tablet_manager.cc:1367] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.051s	user 0.009s	sys 0.000s
I20240501 08:10:45.580608  3954 log.cc:824] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:45.582593  3955 tablet_bootstrap.cc:492] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:45.582901  3955 ts_tablet_manager.cc:1367] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.035s	user 0.021s	sys 0.011s
I20240501 08:10:45.583343  3953 tablet_bootstrap.cc:492] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:45.586006  3959 tablet_bootstrap.cc:492] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:45.586249  3953 tablet_bootstrap.cc:654] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.587551  3953 log.cc:824] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:45.590481  3951 raft_consensus.cc:348] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.591416  3955 raft_consensus.cc:348] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.592259  3955 raft_consensus.cc:374] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.592423  3955 raft_consensus.cc:729] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.593165  3955 consensus_queue.cc:260] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.594740  3954 tablet_bootstrap.cc:492] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:45.595141  3954 ts_tablet_manager.cc:1367] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.027s	user 0.007s	sys 0.002s
I20240501 08:10:45.596436  3953 tablet_bootstrap.cc:492] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:45.596765  3953 ts_tablet_manager.cc:1367] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.004s
I20240501 08:10:45.597517  3952 raft_consensus.cc:348] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.598008  3952 raft_consensus.cc:374] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.598173  3952 raft_consensus.cc:729] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.598912  3952 consensus_queue.cc:260] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.601589  3955 ts_tablet_manager.cc:1398] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.019s	user 0.011s	sys 0.005s
I20240501 08:10:45.602317  3959 tablet_bootstrap.cc:654] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.603617  3959 log.cc:824] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:45.603857  3951 raft_consensus.cc:374] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.604209  3951 raft_consensus.cc:729] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.608332  3955 tablet_bootstrap.cc:492] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:45.608718  3953 raft_consensus.cc:348] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.610033  3955 tablet_bootstrap.cc:654] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.610082  3951 consensus_queue.cc:260] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.613617  3955 tablet_bootstrap.cc:492] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:45.613796  3955 ts_tablet_manager.cc:1367] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 08:10:45.614827  3955 raft_consensus.cc:348] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:45.615082  3955 raft_consensus.cc:374] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.615260  3959 tablet_bootstrap.cc:492] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:45.615227  3955 raft_consensus.cc:729] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.615590  3959 ts_tablet_manager.cc:1367] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.030s	user 0.008s	sys 0.000s
I20240501 08:10:45.615542  3955 consensus_queue.cc:260] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:45.617516  3954 raft_consensus.cc:348] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.618597  3955 ts_tablet_manager.cc:1398] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240501 08:10:45.620229  3954 raft_consensus.cc:374] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.620523  3954 raft_consensus.cc:729] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.623461  3951 ts_tablet_manager.cc:1398] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.006s
I20240501 08:10:45.624081  3951 tablet_bootstrap.cc:492] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:45.625370  3952 ts_tablet_manager.cc:1398] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.046s	user 0.026s	sys 0.012s
I20240501 08:10:45.625852  3951 tablet_bootstrap.cc:654] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.625852  3952 tablet_bootstrap.cc:492] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:45.626032  3953 raft_consensus.cc:374] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.626225  3953 raft_consensus.cc:729] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.626382  3954 consensus_queue.cc:260] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.627020  3953 consensus_queue.cc:260] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.627678  3952 tablet_bootstrap.cc:654] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.630435  3954 ts_tablet_manager.cc:1398] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.035s	user 0.011s	sys 0.000s
I20240501 08:10:45.635053  3951 tablet_bootstrap.cc:492] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:45.635198  3951 ts_tablet_manager.cc:1367] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240501 08:10:45.636291  3951 raft_consensus.cc:348] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:45.636512  3951 raft_consensus.cc:374] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.636497  3953 ts_tablet_manager.cc:1398] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.040s	user 0.012s	sys 0.005s
I20240501 08:10:45.636611  3951 raft_consensus.cc:729] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.636831  3951 consensus_queue.cc:260] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:45.638094  3951 ts_tablet_manager.cc:1398] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:45.637706  3959 raft_consensus.cc:348] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.639966  3952 tablet_bootstrap.cc:492] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:45.640175  3952 ts_tablet_manager.cc:1367] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.008s
I20240501 08:10:45.640202  3959 raft_consensus.cc:374] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.640717  3959 raft_consensus.cc:729] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.641278  3952 raft_consensus.cc:348] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:45.641535  3952 raft_consensus.cc:374] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.641502  3959 consensus_queue.cc:260] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.641670  3952 raft_consensus.cc:729] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.641930  3952 consensus_queue.cc:260] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:45.642978  3952 ts_tablet_manager.cc:1398] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:45.643167  3959 ts_tablet_manager.cc:1398] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.027s	user 0.010s	sys 0.000s
W20240501 08:10:45.649545  3272 tablet.cc:2369] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:45.655748  3797 tablet.cc:2369] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:45.688077  3969 raft_consensus.cc:482] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:45.688355  3969 raft_consensus.cc:504] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.690047  3969 leader_election.cc:288] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:45.697362  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66bace0bf96429eab97117c872963e1" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:45.697403  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66bace0bf96429eab97117c872963e1" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:45.697813  3226 raft_consensus.cc:2454] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:45.697846  3489 raft_consensus.cc:2454] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:45.698349  3818 leader_election.cc:302] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6deb22ef32e2476cb56efabd79a948c0, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:45.698690  3969 raft_consensus.cc:2781] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:45.698829  3969 raft_consensus.cc:482] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:45.698925  3969 raft_consensus.cc:3037] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:45.700932  3969 raft_consensus.cc:504] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.701603  3969 leader_election.cc:288] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:45.702039  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66bace0bf96429eab97117c872963e1" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:45.702244  3489 raft_consensus.cc:3037] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:45.702623  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66bace0bf96429eab97117c872963e1" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:45.702836  3226 raft_consensus.cc:3037] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:45.705237  3489 raft_consensus.cc:2454] T d66bace0bf96429eab97117c872963e1 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:45.705643  3817 leader_election.cc:302] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6deb22ef32e2476cb56efabd79a948c0, 97336c9b436644a798689a443b2f272e; no voters: 
I20240501 08:10:45.705744  3226 raft_consensus.cc:2454] T d66bace0bf96429eab97117c872963e1 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:45.705961  3969 raft_consensus.cc:2781] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:45.706395  3969 raft_consensus.cc:686] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:45.706883  3969 consensus_queue.cc:237] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:45.711076  3084 catalog_manager.cc:5484] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } }
W20240501 08:10:45.728546  3666 tablet.cc:2369] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:45.789052  3535 tablet.cc:2369] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:45.810436  3971 raft_consensus.cc:482] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:45.810719  3971 raft_consensus.cc:504] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.812402  3971 leader_election.cc:288] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:45.820940  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51a508c09466ebc399b8441cd3f64" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:45.821512  3751 raft_consensus.cc:2454] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:45.821821  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51a508c09466ebc399b8441cd3f64" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:45.822165  3556 leader_election.cc:302] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:45.822266  3358 raft_consensus.cc:2454] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:45.822548  3971 raft_consensus.cc:2781] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:45.822687  3971 raft_consensus.cc:482] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:45.822798  3971 raft_consensus.cc:3037] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:45.824757  3971 raft_consensus.cc:504] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.825300  3971 leader_election.cc:288] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:45.825701  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51a508c09466ebc399b8441cd3f64" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:45.825757  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51a508c09466ebc399b8441cd3f64" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:45.825917  3358 raft_consensus.cc:3037] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:45.825911  3751 raft_consensus.cc:3037] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:45.828951  3358 raft_consensus.cc:2454] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:45.828951  3751 raft_consensus.cc:2454] T b3a51a508c09466ebc399b8441cd3f64 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:45.829311  3556 leader_election.cc:302] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:45.829561  3971 raft_consensus.cc:2781] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:45.829972  3971 raft_consensus.cc:686] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 LEADER]: Becoming Leader. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Running, Role: LEADER
I20240501 08:10:45.830427  3971 consensus_queue.cc:237] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:45.834884  3084 catalog_manager.cc:5484] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132). New cstate: current_term: 1 leader_uuid: "29eea43c04ae49edb64ba8803bf09b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240501 08:10:45.854789  3404 tablet.cc:2369] T b3a51a508c09466ebc399b8441cd3f64 P 1979c6dfa70f42699a57e8210a28b878: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:45.865052  3928 tablet.cc:2369] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:46.070411  3963 raft_consensus.cc:482] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:46.070613  3963 raft_consensus.cc:504] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:46.072294  3963 leader_election.cc:288] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:46.079180  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d31a9a1fe048dda14ce04e0050644e" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:46.079345  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d31a9a1fe048dda14ce04e0050644e" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:46.079437  3489 raft_consensus.cc:2454] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:46.079586  3226 raft_consensus.cc:2454] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:46.079933  3686 leader_election.cc:302] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:46.080350  3963 raft_consensus.cc:2781] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:46.080479  3963 raft_consensus.cc:482] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:46.080580  3963 raft_consensus.cc:3037] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:46.082702  3963 raft_consensus.cc:504] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:46.083262  3963 leader_election.cc:288] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Requested vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:46.083657  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d31a9a1fe048dda14ce04e0050644e" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:46.083761  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d31a9a1fe048dda14ce04e0050644e" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:46.083859  3489 raft_consensus.cc:3037] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:46.084012  3226 raft_consensus.cc:3037] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:46.085958  3226 raft_consensus.cc:2454] T b6d31a9a1fe048dda14ce04e0050644e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:46.086145  3489 raft_consensus.cc:2454] T b6d31a9a1fe048dda14ce04e0050644e P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:46.086408  3687 leader_election.cc:302] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:46.086704  3963 raft_consensus.cc:2781] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:46.087049  3963 raft_consensus.cc:686] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [term 1 LEADER]: Becoming Leader. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Running, Role: LEADER
I20240501 08:10:46.087612  3963 consensus_queue.cc:237] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:46.091866  3084 catalog_manager.cc:5484] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 reported cstate change: term changed from 0 to 1, leader changed from <none> to b526e10277944ab0b4744f9de18a8771 (127.2.248.133). New cstate: current_term: 1 leader_uuid: "b526e10277944ab0b4744f9de18a8771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:46.113217  3969 consensus_queue.cc:1035] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:46.120501  3987 consensus_queue.cc:1035] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:46.316735  3971 consensus_queue.cc:1035] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:46.324651  3971 consensus_queue.cc:1035] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:46.516188  3991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:46.516507  3991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:46.523851  3991 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:10:46.565817  3963 consensus_queue.cc:1035] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:46.572563  3982 consensus_queue.cc:1035] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:46.830804  3818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.129:32925, user_credentials={real_user=slave}} blocked reactor thread for 70865us

INSERT report
    rows total: 2000
    time total: 269.1 ms
  time per row: 0.13455 ms
I20240501 08:10:46.948304  3079 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:46.957633  3338 tablet_service.cc:1467] Processing CreateTablet for tablet c7d7f4724716476990b2303cafa4ade4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.957790  3600 tablet_service.cc:1467] Processing CreateTablet for tablet c7d7f4724716476990b2303cafa4ade4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.958149  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d7f4724716476990b2303cafa4ade4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.958245  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d7f4724716476990b2303cafa4ade4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.958231  3337 tablet_service.cc:1467] Processing CreateTablet for tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.958680  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.963312  4057 tablet_bootstrap.cc:492] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:46.964802  4057 tablet_bootstrap.cc:654] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:46.965044  3731 tablet_service.cc:1467] Processing CreateTablet for tablet 89d4132a23724130b25d6bdf7438c315 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.965224  3206 tablet_service.cc:1467] Processing CreateTablet for tablet 89d4132a23724130b25d6bdf7438c315 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.965495  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d4132a23724130b25d6bdf7438c315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.965621  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d4132a23724130b25d6bdf7438c315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.965719  3469 tablet_service.cc:1467] Processing CreateTablet for tablet 89d4132a23724130b25d6bdf7438c315 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.966100  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d4132a23724130b25d6bdf7438c315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.968638  3862 tablet_service.cc:1467] Processing CreateTablet for tablet c7d7f4724716476990b2303cafa4ade4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.969170  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d7f4724716476990b2303cafa4ade4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.972714  3861 tablet_service.cc:1467] Processing CreateTablet for tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.973310  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.985307  4057 tablet_bootstrap.cc:492] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:46.985507  4057 ts_tablet_manager.cc:1367] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.022s	user 0.008s	sys 0.012s
I20240501 08:10:46.985981  4061 tablet_bootstrap.cc:492] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:46.986495  4057 raft_consensus.cc:348] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:46.986707  4057 raft_consensus.cc:374] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:46.986820  4057 raft_consensus.cc:729] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:46.987075  4057 consensus_queue.cc:260] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:46.987949  4061 tablet_bootstrap.cc:654] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:46.988523  4057 ts_tablet_manager.cc:1398] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:46.989337  4059 tablet_bootstrap.cc:492] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:46.990317  3600 tablet_service.cc:1467] Processing CreateTablet for tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f4a3e284aa5d4b28afe32f8d6938546d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:46.990763  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:46.991719  4059 tablet_bootstrap.cc:654] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:46.998344  4061 tablet_bootstrap.cc:492] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:46.998598  4061 ts_tablet_manager.cc:1367] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.006s
I20240501 08:10:46.999683  4063 tablet_bootstrap.cc:492] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:46.999962  4061 raft_consensus.cc:348] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.000348  4061 raft_consensus.cc:374] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.000517  4061 raft_consensus.cc:729] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.000910  4061 consensus_queue.cc:260] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.002166  4057 tablet_bootstrap.cc:492] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:47.004351  4057 tablet_bootstrap.cc:654] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.006457  4060 tablet_bootstrap.cc:492] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:47.007474  4057 tablet_bootstrap.cc:492] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:47.007658  4057 ts_tablet_manager.cc:1367] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.003s
I20240501 08:10:47.008677  4060 tablet_bootstrap.cc:654] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.008744  4057 raft_consensus.cc:348] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.009007  4057 raft_consensus.cc:374] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.009105  4057 raft_consensus.cc:729] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.009387  4057 consensus_queue.cc:260] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.010042  4057 ts_tablet_manager.cc:1398] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:47.010308  4063 tablet_bootstrap.cc:654] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.012600  4062 tablet_bootstrap.cc:492] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:47.012740  4059 tablet_bootstrap.cc:492] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:47.012928  4059 ts_tablet_manager.cc:1367] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.024s	user 0.003s	sys 0.008s
I20240501 08:10:47.013785  4060 tablet_bootstrap.cc:492] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:47.013983  4060 ts_tablet_manager.cc:1367] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240501 08:10:47.013924  4059 raft_consensus.cc:348] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.014428  4062 tablet_bootstrap.cc:654] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.014925  4059 raft_consensus.cc:374] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.014973  4060 raft_consensus.cc:348] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.015117  4059 raft_consensus.cc:729] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.015262  4060 raft_consensus.cc:374] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.015450  4060 raft_consensus.cc:729] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.015549  4059 consensus_queue.cc:260] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.017344  4059 ts_tablet_manager.cc:1398] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:10:47.017871  4061 ts_tablet_manager.cc:1398] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.004s
I20240501 08:10:47.018242  4061 tablet_bootstrap.cc:492] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:47.020512  4061 tablet_bootstrap.cc:654] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.021818  4060 consensus_queue.cc:260] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.022722  4061 tablet_bootstrap.cc:492] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:47.022832  4060 ts_tablet_manager.cc:1398] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240501 08:10:47.023048  4063 tablet_bootstrap.cc:492] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:47.023039  4061 ts_tablet_manager.cc:1367] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240501 08:10:47.023236  4063 ts_tablet_manager.cc:1367] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.024s	user 0.005s	sys 0.000s
I20240501 08:10:47.023857  4061 raft_consensus.cc:348] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.024174  4061 raft_consensus.cc:374] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.024317  4061 raft_consensus.cc:729] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.024456  4062 tablet_bootstrap.cc:492] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:47.024348  4063 raft_consensus.cc:348] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.024636  4062 ts_tablet_manager.cc:1367] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.010s
I20240501 08:10:47.024638  4063 raft_consensus.cc:374] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.024734  4063 raft_consensus.cc:729] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.024672  4061 consensus_queue.cc:260] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.024974  4063 consensus_queue.cc:260] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.025172  4061 ts_tablet_manager.cc:1398] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:10:47.025583  4063 ts_tablet_manager.cc:1398] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:47.025656  4062 raft_consensus.cc:348] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.025861  4062 raft_consensus.cc:374] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.025892  4063 tablet_bootstrap.cc:492] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:47.025941  4062 raft_consensus.cc:729] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.026185  4062 consensus_queue.cc:260] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.026829  4062 ts_tablet_manager.cc:1398] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:47.028193  4063 tablet_bootstrap.cc:654] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.030381  4063 tablet_bootstrap.cc:492] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:47.030522  4063 ts_tablet_manager.cc:1367] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 08:10:47.031235  4063 raft_consensus.cc:348] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.031422  4063 raft_consensus.cc:374] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.031499  4063 raft_consensus.cc:729] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.031731  4063 consensus_queue.cc:260] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.032155  4063 ts_tablet_manager.cc:1398] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:10:47.061152  3966 raft_consensus.cc:482] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:47.061328  3966 raft_consensus.cc:504] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.062842  3966 leader_election.cc:288] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:47.069932  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4132a23724130b25d6bdf7438c315" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:47.070192  3751 raft_consensus.cc:2454] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:47.070641  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4132a23724130b25d6bdf7438c315" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:47.070713  3425 leader_election.cc:302] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:47.070837  3226 raft_consensus.cc:2454] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:47.071087  3966 raft_consensus.cc:2781] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:47.071219  3966 raft_consensus.cc:482] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:47.071343  3966 raft_consensus.cc:3037] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.073542  3966 raft_consensus.cc:504] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.074201  3966 leader_election.cc:288] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:47.074582  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4132a23724130b25d6bdf7438c315" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:47.074622  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d4132a23724130b25d6bdf7438c315" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:47.074782  3751 raft_consensus.cc:3037] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.074797  3226 raft_consensus.cc:3037] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.076990  3751 raft_consensus.cc:2454] T 89d4132a23724130b25d6bdf7438c315 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:47.077030  3226 raft_consensus.cc:2454] T 89d4132a23724130b25d6bdf7438c315 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:47.077379  3425 leader_election.cc:302] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:47.077708  3966 raft_consensus.cc:2781] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:47.078244  3966 raft_consensus.cc:686] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [term 1 LEADER]: Becoming Leader. State: Replica: 97336c9b436644a798689a443b2f272e, State: Running, Role: LEADER
I20240501 08:10:47.078760  3966 consensus_queue.cc:237] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:47.082309  3079 catalog_manager.cc:5484] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97336c9b436644a798689a443b2f272e (127.2.248.131). New cstate: current_term: 1 leader_uuid: "97336c9b436644a798689a443b2f272e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:47.104655  4040 raft_consensus.cc:482] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:47.104805  4040 raft_consensus.cc:504] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.105535  4040 leader_election.cc:288] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:47.111904  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafcf5e6c2af4df9aaf4107dcd5c9b7a" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:47.111891  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafcf5e6c2af4df9aaf4107dcd5c9b7a" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:47.112174  3357 raft_consensus.cc:2454] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:47.112285  3620 raft_consensus.cc:2454] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:47.112622  3818 leader_election.cc:302] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:47.112921  4040 raft_consensus.cc:2781] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:47.113031  4040 raft_consensus.cc:482] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:47.113119  4040 raft_consensus.cc:3037] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.114969  4040 raft_consensus.cc:504] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.115518  4040 leader_election.cc:288] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:47.115842  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafcf5e6c2af4df9aaf4107dcd5c9b7a" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:47.115957  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafcf5e6c2af4df9aaf4107dcd5c9b7a" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:47.116062  3357 raft_consensus.cc:3037] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.116159  3620 raft_consensus.cc:3037] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.118887  3620 raft_consensus.cc:2454] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:47.118887  3357 raft_consensus.cc:2454] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:47.119261  3817 leader_election.cc:302] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:47.119580  4040 raft_consensus.cc:2781] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:47.119699  4040 raft_consensus.cc:686] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:47.119961  4040 consensus_queue.cc:237] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.122452  3079 catalog_manager.cc:5484] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:47.176616  4040 raft_consensus.cc:482] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:47.176800  4040 raft_consensus.cc:504] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.177381  4040 leader_election.cc:288] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:47.177821  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d7f4724716476990b2303cafa4ade4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:47.177872  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d7f4724716476990b2303cafa4ade4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:47.178059  3357 raft_consensus.cc:2454] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:47.178078  3620 raft_consensus.cc:2454] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:47.178458  3817 leader_election.cc:302] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:47.178710  4040 raft_consensus.cc:2781] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:47.178788  4040 raft_consensus.cc:482] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:47.178910  4040 raft_consensus.cc:3037] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.180795  4040 raft_consensus.cc:504] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.181316  4040 leader_election.cc:288] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:47.181684  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d7f4724716476990b2303cafa4ade4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:47.181711  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d7f4724716476990b2303cafa4ade4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:47.181843  3357 raft_consensus.cc:3037] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.181895  3620 raft_consensus.cc:3037] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:47.183595  3357 raft_consensus.cc:2454] T c7d7f4724716476990b2303cafa4ade4 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:47.183916  3818 leader_election.cc:302] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:47.184125  3620 raft_consensus.cc:2454] T c7d7f4724716476990b2303cafa4ade4 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:47.184170  4040 raft_consensus.cc:2781] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:47.184298  4040 raft_consensus.cc:686] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:47.184762  4040 consensus_queue.cc:237] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:47.187310  3079 catalog_manager.cc:5484] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:47.518983  3966 consensus_queue.cc:1035] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:47.525024  3966 consensus_queue.cc:1035] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:47.538249  4083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:47.538539  4083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:47.546015  4083 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:10:47.574977  4040 consensus_queue.cc:1035] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:47.591809  4112 consensus_queue.cc:1035] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:47.625216  4112 consensus_queue.cc:1035] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:47.631183  4112 consensus_queue.cc:1035] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 302.51 ms
  time per row: 0.151255 ms
I20240501 08:10:47.999619  3083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:48.008744  3469 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5d8eded44d4c85b097e7d3cfa153eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.008955  3206 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5d8eded44d4c85b097e7d3cfa153eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.009208  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5d8eded44d4c85b097e7d3cfa153eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.009325  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5d8eded44d4c85b097e7d3cfa153eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.009205  3861 tablet_service.cc:1467] Processing CreateTablet for tablet 500ec289b2d84a6989c1745fab74c5d4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.009619  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500ec289b2d84a6989c1745fab74c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.009932  3731 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5d8eded44d4c85b097e7d3cfa153eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.010324  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5d8eded44d4c85b097e7d3cfa153eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.012952  3600 tablet_service.cc:1467] Processing CreateTablet for tablet 500ec289b2d84a6989c1745fab74c5d4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.013363  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500ec289b2d84a6989c1745fab74c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.016829  3468 tablet_service.cc:1467] Processing CreateTablet for tablet d6c7063056a34e33a2b21f1934910246 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.017359  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c7063056a34e33a2b21f1934910246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.021126  3338 tablet_service.cc:1467] Processing CreateTablet for tablet 500ec289b2d84a6989c1745fab74c5d4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.021342  3730 tablet_service.cc:1467] Processing CreateTablet for tablet d6c7063056a34e33a2b21f1934910246 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.021548  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500ec289b2d84a6989c1745fab74c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.021813  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c7063056a34e33a2b21f1934910246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.024601  3205 tablet_service.cc:1467] Processing CreateTablet for tablet d6c7063056a34e33a2b21f1934910246 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46388a3496964cba9f7736df94284716]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.025079  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c7063056a34e33a2b21f1934910246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.027649  4147 tablet_bootstrap.cc:492] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:48.029320  4147 tablet_bootstrap.cc:654] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.032105  4148 tablet_bootstrap.cc:492] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:48.035873  4148 tablet_bootstrap.cc:654] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.036695  4150 tablet_bootstrap.cc:492] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:48.036965  4147 tablet_bootstrap.cc:492] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:48.037581  4147 ts_tablet_manager.cc:1367] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240501 08:10:48.038558  4147 raft_consensus.cc:348] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.039849  4150 tablet_bootstrap.cc:654] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.040685  4147 raft_consensus.cc:374] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.041061  4147 raft_consensus.cc:729] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.041600  4145 tablet_bootstrap.cc:492] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:48.041836  4147 consensus_queue.cc:260] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.042883  4146 tablet_bootstrap.cc:492] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:48.043462  4144 tablet_bootstrap.cc:492] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:48.044231  4150 tablet_bootstrap.cc:492] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:48.044454  4150 ts_tablet_manager.cc:1367] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240501 08:10:48.045231  4150 raft_consensus.cc:348] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.045500  4144 tablet_bootstrap.cc:654] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.045536  4150 raft_consensus.cc:374] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.045670  4150 raft_consensus.cc:729] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.045727  4147 ts_tablet_manager.cc:1398] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240501 08:10:48.046020  4145 tablet_bootstrap.cc:654] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.045975  4150 consensus_queue.cc:260] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.047160  4146 tablet_bootstrap.cc:654] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.047132  4040 raft_consensus.cc:482] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.047295  4040 raft_consensus.cc:504] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.047940  4040 leader_election.cc:288] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:48.048329  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500ec289b2d84a6989c1745fab74c5d4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:48.048530  3620 raft_consensus.cc:2454] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:48.049017  3817 leader_election.cc:302] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:48.049007  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500ec289b2d84a6989c1745fab74c5d4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:48.049392  4040 raft_consensus.cc:2781] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.049477  4040 raft_consensus.cc:482] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.049598  4040 raft_consensus.cc:3037] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.049890  4148 tablet_bootstrap.cc:492] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
W20240501 08:10:48.050001  3818 leader_election.cc:341] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:10:48.050069  4148 ts_tablet_manager.cc:1367] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.000s
I20240501 08:10:48.050853  4150 ts_tablet_manager.cc:1398] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 08:10:48.051478  4148 raft_consensus.cc:348] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.051723  4148 raft_consensus.cc:374] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.051862  4148 raft_consensus.cc:729] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.052227  4148 consensus_queue.cc:260] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.052747  4040 raft_consensus.cc:504] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.055776  4148 ts_tablet_manager.cc:1398] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 08:10:48.055930  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500ec289b2d84a6989c1745fab74c5d4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:48.056196  4146 tablet_bootstrap.cc:492] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:48.056185  3620 raft_consensus.cc:3037] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.056345  4146 ts_tablet_manager.cc:1367] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.003s
I20240501 08:10:48.057008  4145 tablet_bootstrap.cc:492] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:48.057163  4145 ts_tablet_manager.cc:1367] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.000s
I20240501 08:10:48.057128  4146 raft_consensus.cc:348] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.057327  4146 raft_consensus.cc:374] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.057420  4146 raft_consensus.cc:729] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.057926  4145 raft_consensus.cc:348] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.058212  4146 consensus_queue.cc:260] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.058945  4145 raft_consensus.cc:374] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.059121  4146 ts_tablet_manager.cc:1398] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:48.059121  4145 raft_consensus.cc:729] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.059463  4146 tablet_bootstrap.cc:492] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:48.059355  4145 consensus_queue.cc:260] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.059989  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500ec289b2d84a6989c1745fab74c5d4" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:48.060289  3358 raft_consensus.cc:3037] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.061695  4146 tablet_bootstrap.cc:654] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.062994  3358 raft_consensus.cc:2454] T 500ec289b2d84a6989c1745fab74c5d4 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:48.063468  3818 leader_election.cc:302] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:48.063802  4044 raft_consensus.cc:2781] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:48.063867  4040 leader_election.cc:288] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:48.063988  4044 raft_consensus.cc:686] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:48.064390  4044 consensus_queue.cc:237] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.064967  4144 tablet_bootstrap.cc:492] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:48.065101  4144 ts_tablet_manager.cc:1367] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.022s	user 0.003s	sys 0.004s
I20240501 08:10:48.065387  3620 raft_consensus.cc:2454] T 500ec289b2d84a6989c1745fab74c5d4 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:48.065904  4144 raft_consensus.cc:348] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.066139  4144 raft_consensus.cc:374] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.066274  4144 raft_consensus.cc:729] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.066541  4144 consensus_queue.cc:260] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.067103  4144 ts_tablet_manager.cc:1398] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:10:48.067617  4145 ts_tablet_manager.cc:1398] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20240501 08:10:48.067940  4146 tablet_bootstrap.cc:492] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:48.068074  4144 tablet_bootstrap.cc:492] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:48.068158  4146 ts_tablet_manager.cc:1367] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240501 08:10:48.068600  4145 tablet_bootstrap.cc:492] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:48.069010  4146 raft_consensus.cc:348] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.069289  4146 raft_consensus.cc:374] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.069401  4146 raft_consensus.cc:729] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.069662  4146 consensus_queue.cc:260] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.069896  4144 tablet_bootstrap.cc:654] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.070243  4146 ts_tablet_manager.cc:1398] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:10:48.070655  3083 catalog_manager.cc:5484] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:48.072265  4145 tablet_bootstrap.cc:654] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.077234  4145 tablet_bootstrap.cc:492] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:48.077394  4144 tablet_bootstrap.cc:492] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:48.077432  4145 ts_tablet_manager.cc:1367] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.004s
I20240501 08:10:48.077546  4144 ts_tablet_manager.cc:1367] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.008s
I20240501 08:10:48.077991  4145 raft_consensus.cc:348] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.078158  4145 raft_consensus.cc:374] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.078119  4144 raft_consensus.cc:348] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.078233  4145 raft_consensus.cc:729] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.078336  4144 raft_consensus.cc:374] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.078425  4144 raft_consensus.cc:729] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.078510  4145 consensus_queue.cc:260] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.078696  4144 consensus_queue.cc:260] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.079002  4145 ts_tablet_manager.cc:1398] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240501 08:10:48.079615  4144 ts_tablet_manager.cc:1398] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:10:48.109464  4156 raft_consensus.cc:482] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.109669  4156 raft_consensus.cc:504] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.111092  4156 leader_election.cc:288] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:48.117871  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5d8eded44d4c85b097e7d3cfa153eb" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:48.118104  3489 raft_consensus.cc:2454] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:48.118137  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5d8eded44d4c85b097e7d3cfa153eb" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:48.118386  3750 raft_consensus.cc:2454] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:48.118609  3161 leader_election.cc:302] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:48.118947  4156 raft_consensus.cc:2781] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.119091  4156 raft_consensus.cc:482] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.119186  4156 raft_consensus.cc:3037] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.121063  4156 raft_consensus.cc:504] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.121589  4156 leader_election.cc:288] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:48.121956  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5d8eded44d4c85b097e7d3cfa153eb" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:48.122004  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5d8eded44d4c85b097e7d3cfa153eb" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:48.122114  3750 raft_consensus.cc:3037] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.122148  3489 raft_consensus.cc:3037] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.123811  3750 raft_consensus.cc:2454] T 5c5d8eded44d4c85b097e7d3cfa153eb P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:10:48.124008  3489 raft_consensus.cc:2454] T 5c5d8eded44d4c85b097e7d3cfa153eb P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:10:48.124171  3162 leader_election.cc:302] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:48.124440  4156 raft_consensus.cc:2781] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:48.124801  4156 raft_consensus.cc:686] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 LEADER]: Becoming Leader. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Running, Role: LEADER
I20240501 08:10:48.125247  4156 consensus_queue.cc:237] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.128433  3076 catalog_manager.cc:5484] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:48.188042  4137 raft_consensus.cc:482] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.188270  4137 raft_consensus.cc:504] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.189170  4137 leader_election.cc:288] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:48.189673  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c7063056a34e33a2b21f1934910246" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:48.189736  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c7063056a34e33a2b21f1934910246" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:48.189888  3750 raft_consensus.cc:2454] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:48.189994  3226 raft_consensus.cc:2454] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:48.190299  3425 leader_election.cc:302] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:48.190536  4137 raft_consensus.cc:2781] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.190659  4137 raft_consensus.cc:482] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.190761  4137 raft_consensus.cc:3037] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.193172  4137 raft_consensus.cc:504] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.193804  4137 leader_election.cc:288] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:48.194209  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c7063056a34e33a2b21f1934910246" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:48.194240  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c7063056a34e33a2b21f1934910246" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:48.194437  3750 raft_consensus.cc:3037] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.194437  3226 raft_consensus.cc:3037] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.197187  3750 raft_consensus.cc:2454] T d6c7063056a34e33a2b21f1934910246 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:48.197187  3226 raft_consensus.cc:2454] T d6c7063056a34e33a2b21f1934910246 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:48.197605  3425 leader_election.cc:302] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:48.197906  4137 raft_consensus.cc:2781] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:48.198086  4137 raft_consensus.cc:686] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [term 1 LEADER]: Becoming Leader. State: Replica: 97336c9b436644a798689a443b2f272e, State: Running, Role: LEADER
I20240501 08:10:48.198431  4137 consensus_queue.cc:237] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:48.201567  3076 catalog_manager.cc:5484] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97336c9b436644a798689a443b2f272e (127.2.248.131). New cstate: current_term: 1 leader_uuid: "97336c9b436644a798689a443b2f272e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:48.513031  4156 consensus_queue.cc:1035] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.518988  4156 consensus_queue.cc:1035] T 5c5d8eded44d4c85b097e7d3cfa153eb P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:48.540346  4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:48.540634  4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:48.547960  4167 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:48.559406  4044 consensus_queue.cc:1035] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.564649  4044 consensus_queue.cc:1035] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 08:10:48.639340  4137 consensus_queue.cc:1035] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.657269  3966 consensus_queue.cc:1035] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 244.948 ms
  time per row: 0.122474 ms
I20240501 08:10:48.942056  3083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:48.950794  3338 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc4623bbc0e4a20944a3f863e311128 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.951267  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc4623bbc0e4a20944a3f863e311128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.951323  3861 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc4623bbc0e4a20944a3f863e311128 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.951681  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc4623bbc0e4a20944a3f863e311128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.951778  3337 tablet_service.cc:1467] Processing CreateTablet for tablet 75ed8727af504012bf9d5242deec4fd6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.952216  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ed8727af504012bf9d5242deec4fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.952442  3862 tablet_service.cc:1467] Processing CreateTablet for tablet 75ed8727af504012bf9d5242deec4fd6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.952589  3599 tablet_service.cc:1467] Processing CreateTablet for tablet 75ed8727af504012bf9d5242deec4fd6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.952760  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ed8727af504012bf9d5242deec4fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.952976  3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ed8727af504012bf9d5242deec4fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.956748  3600 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc4623bbc0e4a20944a3f863e311128 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.956903  3206 tablet_service.cc:1467] Processing CreateTablet for tablet 709f953859ea4186b38c911c30b5f9b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.957553  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc4623bbc0e4a20944a3f863e311128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.957800  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709f953859ea4186b38c911c30b5f9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.961773  3469 tablet_service.cc:1467] Processing CreateTablet for tablet 709f953859ea4186b38c911c30b5f9b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.962296  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709f953859ea4186b38c911c30b5f9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.964042  4228 tablet_bootstrap.cc:492] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:48.964773  3731 tablet_service.cc:1467] Processing CreateTablet for tablet 709f953859ea4186b38c911c30b5f9b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e3193237a9f3498588cccf3b3ae4fecd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:48.965167  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709f953859ea4186b38c911c30b5f9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:48.965731  4228 tablet_bootstrap.cc:654] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.971496  4228 tablet_bootstrap.cc:492] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:48.972254  4228 ts_tablet_manager.cc:1367] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240501 08:10:48.973568  4228 raft_consensus.cc:348] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:48.975069  4228 raft_consensus.cc:374] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.975425  4232 tablet_bootstrap.cc:492] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:48.975417  4228 raft_consensus.cc:729] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.976251  4228 consensus_queue.cc:260] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:48.977556  4228 ts_tablet_manager.cc:1398] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:10:48.977597  4232 tablet_bootstrap.cc:654] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.978474  4228 tablet_bootstrap.cc:492] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:48.980221  4228 tablet_bootstrap.cc:654] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.987835  4228 tablet_bootstrap.cc:492] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:48.987977  4228 ts_tablet_manager.cc:1367] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.004s
I20240501 08:10:48.988651  4232 tablet_bootstrap.cc:492] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:48.988844  4232 ts_tablet_manager.cc:1367] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.000s
I20240501 08:10:48.989468  4228 raft_consensus.cc:348] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.989584  4232 raft_consensus.cc:348] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:48.989897  4232 raft_consensus.cc:374] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.990052  4232 raft_consensus.cc:729] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.990410  4228 raft_consensus.cc:374] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.990628  4228 raft_consensus.cc:729] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.991005  4228 consensus_queue.cc:260] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:48.991446  4232 consensus_queue.cc:260] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:48.992277  4232 ts_tablet_manager.cc:1398] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:48.993130  4228 ts_tablet_manager.cc:1398] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240501 08:10:48.999733  4233 tablet_bootstrap.cc:492] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:49.001048  4236 tablet_bootstrap.cc:492] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:49.003660  4236 tablet_bootstrap.cc:654] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.006345  4233 tablet_bootstrap.cc:654] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.007975  4231 tablet_bootstrap.cc:492] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:49.010496  4236 tablet_bootstrap.cc:492] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:49.010694  4236 ts_tablet_manager.cc:1367] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.003s
I20240501 08:10:49.011225  4229 tablet_bootstrap.cc:492] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:49.011621  4236 raft_consensus.cc:348] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:49.011796  4236 raft_consensus.cc:374] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.011912  4236 raft_consensus.cc:729] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.012173  4236 consensus_queue.cc:260] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:49.012398  4231 tablet_bootstrap.cc:654] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.012801  4236 ts_tablet_manager.cc:1398] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:49.012952  4229 tablet_bootstrap.cc:654] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.014219  4231 tablet_bootstrap.cc:492] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:49.014396  4231 ts_tablet_manager.cc:1367] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240501 08:10:49.015776  4233 tablet_bootstrap.cc:492] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:49.015944  4229 tablet_bootstrap.cc:492] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:49.015812  4231 raft_consensus.cc:348] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:49.016022  4231 raft_consensus.cc:374] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.015991  4233 ts_tablet_manager.cc:1367] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.000s
I20240501 08:10:49.016136  4229 ts_tablet_manager.cc:1367] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.005s
I20240501 08:10:49.016220  4231 raft_consensus.cc:729] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.016577  4231 consensus_queue.cc:260] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:49.016885  4229 raft_consensus.cc:348] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:49.017127  4229 raft_consensus.cc:374] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.016999  4233 raft_consensus.cc:348] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:49.017261  4233 raft_consensus.cc:374] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.017249  4229 raft_consensus.cc:729] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.017359  4231 ts_tablet_manager.cc:1398] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:49.017480  4229 consensus_queue.cc:260] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:49.018011  4229 ts_tablet_manager.cc:1398] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:10:49.017433  4233 raft_consensus.cc:729] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.018307  4233 consensus_queue.cc:260] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:49.018764  4233 ts_tablet_manager.cc:1398] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240501 08:10:49.019001  4233 tablet_bootstrap.cc:492] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:49.020390  4229 tablet_bootstrap.cc:492] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:49.020545  4233 tablet_bootstrap.cc:654] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.022536  4233 tablet_bootstrap.cc:492] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:49.022585  4229 tablet_bootstrap.cc:654] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.022760  4233 ts_tablet_manager.cc:1367] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 08:10:49.023582  4233 raft_consensus.cc:348] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:49.023799  4233 raft_consensus.cc:374] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.023872  4233 raft_consensus.cc:729] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.024092  4233 consensus_queue.cc:260] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:49.024571  4229 tablet_bootstrap.cc:492] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:49.024682  4233 ts_tablet_manager.cc:1398] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:10:49.024721  4229 ts_tablet_manager.cc:1367] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:10:49.025336  4229 raft_consensus.cc:348] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:49.025571  4229 raft_consensus.cc:374] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.025655  4229 raft_consensus.cc:729] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.025914  4229 consensus_queue.cc:260] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:49.026345  4229 ts_tablet_manager.cc:1398] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:10:49.303846  3971 raft_consensus.cc:482] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:49.304034  3971 raft_consensus.cc:504] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:49.304764  3971 leader_election.cc:288] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:10:49.305307  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc4623bbc0e4a20944a3f863e311128" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:49.305624  3358 raft_consensus.cc:2454] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:49.306097  3556 leader_election.cc:302] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:49.306437  3971 raft_consensus.cc:2781] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:49.306551  3971 raft_consensus.cc:482] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:49.306659  3971 raft_consensus.cc:3037] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.309427  3971 raft_consensus.cc:504] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:49.310180  3971 leader_election.cc:288] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:10:49.310508  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc4623bbc0e4a20944a3f863e311128" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:49.310724  3358 raft_consensus.cc:3037] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.312085  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc4623bbc0e4a20944a3f863e311128" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0" is_pre_election: true
I20240501 08:10:49.312101  3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc4623bbc0e4a20944a3f863e311128" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0"
I20240501 08:10:49.312469  3882 raft_consensus.cc:2454] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:49.312909  3358 raft_consensus.cc:2454] T 5bc4623bbc0e4a20944a3f863e311128 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:49.313271  3556 leader_election.cc:302] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 6deb22ef32e2476cb56efabd79a948c0
I20240501 08:10:49.313581  3971 raft_consensus.cc:2781] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.314049  3971 raft_consensus.cc:686] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 LEADER]: Becoming Leader. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Running, Role: LEADER
I20240501 08:10:49.314400  3971 consensus_queue.cc:237] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:49.316912  4112 raft_consensus.cc:482] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:49.317076  4112 raft_consensus.cc:504] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:49.317230  3076 catalog_manager.cc:5484] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132). New cstate: current_term: 1 leader_uuid: "29eea43c04ae49edb64ba8803bf09b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:49.317739  4112 leader_election.cc:288] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:49.318141  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ed8727af504012bf9d5242deec4fd6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:49.318346  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ed8727af504012bf9d5242deec4fd6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:49.318418  3620 raft_consensus.cc:2454] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:49.318603  3358 raft_consensus.cc:2454] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:49.319056  3818 leader_election.cc:302] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:49.319357  4112 raft_consensus.cc:2781] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:49.319453  4112 raft_consensus.cc:482] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:49.319562  4112 raft_consensus.cc:3037] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.321597  4112 raft_consensus.cc:504] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:49.322193  4112 leader_election.cc:288] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:49.322563  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ed8727af504012bf9d5242deec4fd6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:49.322700  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ed8727af504012bf9d5242deec4fd6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:49.322777  3620 raft_consensus.cc:3037] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.322885  3358 raft_consensus.cc:3037] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.325423  3620 raft_consensus.cc:2454] T 75ed8727af504012bf9d5242deec4fd6 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:49.326047  3358 raft_consensus.cc:2454] T 75ed8727af504012bf9d5242deec4fd6 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:49.326053  3817 leader_election.cc:302] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:49.326355  4112 raft_consensus.cc:2781] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.326711  4112 raft_consensus.cc:686] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:49.326936  4112 consensus_queue.cc:237] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:49.329406  3076 catalog_manager.cc:5484] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:49.334172  4129 raft_consensus.cc:482] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:49.334348  4129 raft_consensus.cc:504] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:49.334910  4129 leader_election.cc:288] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:49.335402  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709f953859ea4186b38c911c30b5f9b0" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:49.335433  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709f953859ea4186b38c911c30b5f9b0" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:49.335640  3226 raft_consensus.cc:2454] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:49.335660  3489 raft_consensus.cc:2454] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:49.336037  3686 leader_election.cc:302] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:49.336328  4129 raft_consensus.cc:2781] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:49.336429  4129 raft_consensus.cc:482] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:49.336511  4129 raft_consensus.cc:3037] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.338644  4129 raft_consensus.cc:504] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:49.339298  4129 leader_election.cc:288] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Requested vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:49.339740  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709f953859ea4186b38c911c30b5f9b0" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:49.339763  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709f953859ea4186b38c911c30b5f9b0" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:49.339913  3226 raft_consensus.cc:3037] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.339947  3489 raft_consensus.cc:3037] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.341704  3226 raft_consensus.cc:2454] T 709f953859ea4186b38c911c30b5f9b0 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:49.341922  3489 raft_consensus.cc:2454] T 709f953859ea4186b38c911c30b5f9b0 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:49.342034  3687 leader_election.cc:302] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:49.342319  4129 raft_consensus.cc:2781] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.342664  4129 raft_consensus.cc:686] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [term 1 LEADER]: Becoming Leader. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Running, Role: LEADER
I20240501 08:10:49.342957  4129 consensus_queue.cc:237] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:49.345853  3076 catalog_manager.cc:5484] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 reported cstate change: term changed from 0 to 1, leader changed from <none> to b526e10277944ab0b4744f9de18a8771 (127.2.248.133). New cstate: current_term: 1 leader_uuid: "b526e10277944ab0b4744f9de18a8771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } }
W20240501 08:10:49.751907  4251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:49.752310  4251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:49.760277  4251 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:49.761581  3881 raft_consensus.cc:3037] T 5bc4623bbc0e4a20944a3f863e311128 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.780695  4129 consensus_queue.cc:1035] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:49.795490  4249 consensus_queue.cc:1035] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 08:10:49.809221  4112 consensus_queue.cc:1035] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:49.818120  4247 consensus_queue.cc:1035] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:49.847576  3971 consensus_queue.cc:1035] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:49.889827  3971 consensus_queue.cc:1035] T 5bc4623bbc0e4a20944a3f863e311128 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 283.747 ms
  time per row: 0.141873 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:10:51.162603  3076 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:51.171931  3469 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc5921f69664791a6d20c48440e6bf9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.172675  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc5921f69664791a6d20c48440e6bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.172737  3862 tablet_service.cc:1467] Processing CreateTablet for tablet ad0a4d86e4a145b3a45f43e6b2ffc7da (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.173120  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0a4d86e4a145b3a45f43e6b2ffc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.173418  3731 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc5921f69664791a6d20c48440e6bf9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.173832  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc5921f69664791a6d20c48440e6bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.174113  3599 tablet_service.cc:1467] Processing CreateTablet for tablet ad0a4d86e4a145b3a45f43e6b2ffc7da (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.174510  3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0a4d86e4a145b3a45f43e6b2ffc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.177426  3206 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc5921f69664791a6d20c48440e6bf9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.177444  3337 tablet_service.cc:1467] Processing CreateTablet for tablet ad0a4d86e4a145b3a45f43e6b2ffc7da (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.177824  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc5921f69664791a6d20c48440e6bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.177889  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad0a4d86e4a145b3a45f43e6b2ffc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.180454  3205 tablet_service.cc:1467] Processing CreateTablet for tablet f6212c5cba30496c80c61a4014be2b9b (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.180891  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6212c5cba30496c80c61a4014be2b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.184952  3730 tablet_service.cc:1467] Processing CreateTablet for tablet f6212c5cba30496c80c61a4014be2b9b (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.185170  3468 tablet_service.cc:1467] Processing CreateTablet for tablet f6212c5cba30496c80c61a4014be2b9b (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c1f06afb3cf4891b93d460ec39f5c39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.185425  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6212c5cba30496c80c61a4014be2b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.185542  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6212c5cba30496c80c61a4014be2b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.200795  4329 tablet_bootstrap.cc:492] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:51.204155  4330 tablet_bootstrap.cc:492] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:51.205142  4331 tablet_bootstrap.cc:492] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:51.207136  4328 tablet_bootstrap.cc:492] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:51.207264  4327 tablet_bootstrap.cc:492] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:51.208055  4331 tablet_bootstrap.cc:654] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.208711  4330 tablet_bootstrap.cc:654] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.209585  4328 tablet_bootstrap.cc:654] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.209692  4327 tablet_bootstrap.cc:654] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.210942  4329 tablet_bootstrap.cc:654] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.213745  4326 tablet_bootstrap.cc:492] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:51.214694  4330 tablet_bootstrap.cc:492] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:51.214867  4327 tablet_bootstrap.cc:492] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:51.214859  4330 ts_tablet_manager.cc:1367] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240501 08:10:51.215010  4327 ts_tablet_manager.cc:1367] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.004s
I20240501 08:10:51.215250  4331 tablet_bootstrap.cc:492] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:51.215386  4331 ts_tablet_manager.cc:1367] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.007s
I20240501 08:10:51.215699  4330 raft_consensus.cc:348] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.215788  4327 raft_consensus.cc:348] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.215943  4330 raft_consensus.cc:374] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.215984  4327 raft_consensus.cc:374] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.215948  4331 raft_consensus.cc:348] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.216197  4331 raft_consensus.cc:374] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.216336  4331 raft_consensus.cc:729] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.216449  4327 raft_consensus.cc:729] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.216722  4327 consensus_queue.cc:260] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.216745  4331 consensus_queue.cc:260] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.217177  4330 raft_consensus.cc:729] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.217595  4330 consensus_queue.cc:260] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.218008  4327 ts_tablet_manager.cc:1398] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:10:51.218986  4327 tablet_bootstrap.cc:492] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:51.218992  4330 ts_tablet_manager.cc:1398] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:10:51.219820  4326 tablet_bootstrap.cc:654] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.222303  4328 tablet_bootstrap.cc:492] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:51.222522  4328 ts_tablet_manager.cc:1367] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.016s	user 0.002s	sys 0.002s
I20240501 08:10:51.223419  4328 raft_consensus.cc:348] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.223690  4328 raft_consensus.cc:374] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.223825  4328 raft_consensus.cc:729] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.224124  4328 consensus_queue.cc:260] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.224529  4331 ts_tablet_manager.cc:1398] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.008s
I20240501 08:10:51.225462  4328 ts_tablet_manager.cc:1398] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:10:51.225813  4328 tablet_bootstrap.cc:492] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:51.227622  4328 tablet_bootstrap.cc:654] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.228665  4327 tablet_bootstrap.cc:654] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.229921  4329 tablet_bootstrap.cc:492] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:51.230134  4329 ts_tablet_manager.cc:1367] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.029s	user 0.005s	sys 0.003s
I20240501 08:10:51.231105  4329 raft_consensus.cc:348] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.231408  4329 raft_consensus.cc:374] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.231514  4329 raft_consensus.cc:729] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.231844  4329 consensus_queue.cc:260] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.232597  4329 ts_tablet_manager.cc:1398] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:10:51.232942  4329 tablet_bootstrap.cc:492] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:51.234998  4329 tablet_bootstrap.cc:654] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.235637  4328 tablet_bootstrap.cc:492] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:51.236109  4328 ts_tablet_manager.cc:1367] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.002s
I20240501 08:10:51.237025  4328 raft_consensus.cc:348] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.237301  4326 tablet_bootstrap.cc:492] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:51.237476  4326 ts_tablet_manager.cc:1367] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.024s	user 0.006s	sys 0.000s
I20240501 08:10:51.237731  4337 raft_consensus.cc:482] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:51.237970  4337 raft_consensus.cc:504] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.239188  4326 raft_consensus.cc:348] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.239429  4326 raft_consensus.cc:374] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.239542  4326 raft_consensus.cc:729] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.239547  4328 raft_consensus.cc:374] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.239689  4328 raft_consensus.cc:729] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.239837  4326 consensus_queue.cc:260] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.239991  4328 consensus_queue.cc:260] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.240690  4328 ts_tablet_manager.cc:1398] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:10:51.243683  4326 ts_tablet_manager.cc:1398] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240501 08:10:51.255153  4327 tablet_bootstrap.cc:492] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:51.255376  4327 ts_tablet_manager.cc:1367] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.036s	user 0.010s	sys 0.003s
I20240501 08:10:51.256436  4327 raft_consensus.cc:348] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.256726  4327 raft_consensus.cc:374] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.256814  4327 raft_consensus.cc:729] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.257117  4327 consensus_queue.cc:260] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.258476  4337 leader_election.cc:288] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:51.258806  4329 tablet_bootstrap.cc:492] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:51.259060  4329 ts_tablet_manager.cc:1367] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.026s	user 0.002s	sys 0.009s
I20240501 08:10:51.260023  4329 raft_consensus.cc:348] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.260411  4329 raft_consensus.cc:374] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.260546  4329 raft_consensus.cc:729] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.260601  4327 ts_tablet_manager.cc:1398] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20240501 08:10:51.260896  4329 consensus_queue.cc:260] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.262728  4329 ts_tablet_manager.cc:1398] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:51.267433  3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0a4d86e4a145b3a45f43e6b2ffc7da" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0" is_pre_election: true
I20240501 08:10:51.267665  3881 raft_consensus.cc:2454] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 0.
I20240501 08:10:51.268170  3291 leader_election.cc:302] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:51.268606  4337 raft_consensus.cc:2781] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:51.268712  4337 raft_consensus.cc:482] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:51.268793  4337 raft_consensus.cc:3037] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.268909  3619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0a4d86e4a145b3a45f43e6b2ffc7da" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:51.269073  3619 raft_consensus.cc:2454] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 0.
I20240501 08:10:51.316100  4137 raft_consensus.cc:482] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:51.316329  4137 raft_consensus.cc:504] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.317040  4137 leader_election.cc:288] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:51.317600  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc5921f69664791a6d20c48440e6bf9" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:51.317656  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc5921f69664791a6d20c48440e6bf9" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:51.317890  3750 raft_consensus.cc:2454] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:51.317914  3226 raft_consensus.cc:2454] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:51.318370  3425 leader_election.cc:302] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:51.318670  4137 raft_consensus.cc:2781] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:51.318806  4137 raft_consensus.cc:482] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:51.318897  4137 raft_consensus.cc:3037] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.323895  4137 raft_consensus.cc:504] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.323921  4337 raft_consensus.cc:504] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.324687  4137 leader_election.cc:288] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:51.325033  4337 leader_election.cc:288] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 election: Requested vote from peers 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:51.325186  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc5921f69664791a6d20c48440e6bf9" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:51.325412  3226 raft_consensus.cc:3037] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.325626  3619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0a4d86e4a145b3a45f43e6b2ffc7da" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:51.325845  3619 raft_consensus.cc:3037] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.326020  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc5921f69664791a6d20c48440e6bf9" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:51.326233  3750 raft_consensus.cc:3037] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.328050  3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0a4d86e4a145b3a45f43e6b2ffc7da" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0"
I20240501 08:10:51.328274  3881 raft_consensus.cc:3037] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.328552  4156 raft_consensus.cc:482] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:51.328716  4156 raft_consensus.cc:504] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.329345  4156 leader_election.cc:288] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:10:51.329770  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6212c5cba30496c80c61a4014be2b9b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:51.329770  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6212c5cba30496c80c61a4014be2b9b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:51.329962  3751 raft_consensus.cc:2454] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:51.329958  3489 raft_consensus.cc:2454] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:51.330330  3162 leader_election.cc:302] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:51.330626  4156 raft_consensus.cc:2781] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:51.330729  4156 raft_consensus.cc:482] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:51.330837  4156 raft_consensus.cc:3037] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.425206  3619 raft_consensus.cc:2454] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 1.
I20240501 08:10:51.425206  3750 raft_consensus.cc:2454] T 8fc5921f69664791a6d20c48440e6bf9 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:51.425837  3425 leader_election.cc:302] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:51.425948  3293 leader_election.cc:302] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:51.426414  4337 raft_consensus.cc:2781] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:51.426438  3881 raft_consensus.cc:2454] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 1.
I20240501 08:10:51.427009  4337 raft_consensus.cc:686] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [term 1 LEADER]: Becoming Leader. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Running, Role: LEADER
I20240501 08:10:51.427618  4337 consensus_queue.cc:237] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:51.427954  3226 raft_consensus.cc:2454] T 8fc5921f69664791a6d20c48440e6bf9 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:51.428162  4156 raft_consensus.cc:504] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.428472  4137 raft_consensus.cc:2781] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:51.428928  4156 leader_election.cc:288] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Requested vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:10:51.430032  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6212c5cba30496c80c61a4014be2b9b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:51.430272  3489 raft_consensus.cc:3037] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.430598  4137 raft_consensus.cc:686] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [term 1 LEADER]: Becoming Leader. State: Replica: 97336c9b436644a798689a443b2f272e, State: Running, Role: LEADER
I20240501 08:10:51.430724  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6212c5cba30496c80c61a4014be2b9b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:51.430994  3750 raft_consensus.cc:3037] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.430943  4137 consensus_queue.cc:237] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:51.431903  3079 catalog_manager.cc:5484] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "1979c6dfa70f42699a57e8210a28b878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:51.435454  3076 catalog_manager.cc:5484] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97336c9b436644a798689a443b2f272e (127.2.248.131). New cstate: current_term: 1 leader_uuid: "97336c9b436644a798689a443b2f272e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:51.524428  3750 raft_consensus.cc:2454] T f6212c5cba30496c80c61a4014be2b9b P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:10:51.524436  3489 raft_consensus.cc:2454] T f6212c5cba30496c80c61a4014be2b9b P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:10:51.525043  3161 leader_election.cc:302] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:51.525401  4156 raft_consensus.cc:2781] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:51.525583  4156 raft_consensus.cc:686] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 LEADER]: Becoming Leader. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Running, Role: LEADER
I20240501 08:10:51.525907  4156 consensus_queue.cc:237] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:51.528613  3076 catalog_manager.cc:5484] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:51.838876  4337 consensus_queue.cc:1035] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:51.847429  4337 consensus_queue.cc:1035] T ad0a4d86e4a145b3a45f43e6b2ffc7da P 1979c6dfa70f42699a57e8210a28b878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:51.914227  4156 consensus_queue.cc:1035] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:51.922971  4346 consensus_queue.cc:1035] T f6212c5cba30496c80c61a4014be2b9b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:51.925172  4352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:51.925562  4352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:10:51.928912  4137 consensus_queue.cc:1035] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:51.934279  4352 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:51.938158  4137 consensus_queue.cc:1035] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 257.271 ms
  time per row: 0.128635 ms
I20240501 08:10:52.367240  3083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:52.375588  3599 tablet_service.cc:1467] Processing CreateTablet for tablet 867e0435e12a4255a9aaa7d42dbe45a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.375883  3862 tablet_service.cc:1467] Processing CreateTablet for tablet 867e0435e12a4255a9aaa7d42dbe45a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.375883  3337 tablet_service.cc:1467] Processing CreateTablet for tablet 867e0435e12a4255a9aaa7d42dbe45a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.376068  3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867e0435e12a4255a9aaa7d42dbe45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.376318  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867e0435e12a4255a9aaa7d42dbe45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.376945  3861 tablet_service.cc:1467] Processing CreateTablet for tablet 01562e1616fb4a44af73d56f6b75b8e6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.377303  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01562e1616fb4a44af73d56f6b75b8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.377589  3731 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae945fb63414d88b8ec336244aeb55e (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.377982  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae945fb63414d88b8ec336244aeb55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.380573  3205 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae945fb63414d88b8ec336244aeb55e (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.380975  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae945fb63414d88b8ec336244aeb55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.381150  3600 tablet_service.cc:1467] Processing CreateTablet for tablet 01562e1616fb4a44af73d56f6b75b8e6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.381502  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01562e1616fb4a44af73d56f6b75b8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.384572  3338 tablet_service.cc:1467] Processing CreateTablet for tablet 01562e1616fb4a44af73d56f6b75b8e6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.389497  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867e0435e12a4255a9aaa7d42dbe45a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.389631  3468 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae945fb63414d88b8ec336244aeb55e (DEFAULT_TABLE table=rebalance_test_table_5 [id=090fc64a8c3d4c02b23d2a67f04e1ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.390054  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae945fb63414d88b8ec336244aeb55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.392429  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01562e1616fb4a44af73d56f6b75b8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.409181  4424 tablet_bootstrap.cc:492] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:52.412211  4424 tablet_bootstrap.cc:654] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.412351  4426 tablet_bootstrap.cc:492] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:52.412940  4425 tablet_bootstrap.cc:492] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:52.413924  4422 tablet_bootstrap.cc:492] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:52.414470  4426 tablet_bootstrap.cc:654] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.414939  4425 tablet_bootstrap.cc:654] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.415714  4422 tablet_bootstrap.cc:654] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.416937  4423 tablet_bootstrap.cc:492] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:52.419203  4428 tablet_bootstrap.cc:492] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:52.419214  4423 tablet_bootstrap.cc:654] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.421799  4428 tablet_bootstrap.cc:654] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.422017  4425 tablet_bootstrap.cc:492] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:52.422235  4425 ts_tablet_manager.cc:1367] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240501 08:10:52.422770  4422 tablet_bootstrap.cc:492] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:52.422914  4422 ts_tablet_manager.cc:1367] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240501 08:10:52.423539  4425 raft_consensus.cc:348] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.423674  4422 raft_consensus.cc:348] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.423861  4422 raft_consensus.cc:374] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.423940  4422 raft_consensus.cc:729] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.424046  4425 raft_consensus.cc:374] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.424257  4425 raft_consensus.cc:729] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.424624  4425 consensus_queue.cc:260] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.425562  4424 tablet_bootstrap.cc:492] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:52.425748  4424 ts_tablet_manager.cc:1367] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.017s	user 0.001s	sys 0.004s
I20240501 08:10:52.426510  4424 raft_consensus.cc:348] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.427667  4424 raft_consensus.cc:374] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.427793  4424 raft_consensus.cc:729] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.428054  4424 consensus_queue.cc:260] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.428951  4425 ts_tablet_manager.cc:1398] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240501 08:10:52.431521  4424 ts_tablet_manager.cc:1398] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.001s
I20240501 08:10:52.432394  4424 tablet_bootstrap.cc:492] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:52.434270  4422 consensus_queue.cc:260] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.434773  4423 tablet_bootstrap.cc:492] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:52.434927  4423 ts_tablet_manager.cc:1367] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.000s
I20240501 08:10:52.435564  4424 tablet_bootstrap.cc:654] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.436295  4422 ts_tablet_manager.cc:1398] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240501 08:10:52.436604  4422 tablet_bootstrap.cc:492] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:52.436908  4426 tablet_bootstrap.cc:492] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:52.437110  4426 ts_tablet_manager.cc:1367] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.025s	user 0.009s	sys 0.011s
I20240501 08:10:52.437191  4423 raft_consensus.cc:348] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.437412  4423 raft_consensus.cc:374] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.437506  4423 raft_consensus.cc:729] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.438611  4426 raft_consensus.cc:348] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.438860  4426 raft_consensus.cc:374] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.438997  4426 raft_consensus.cc:729] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.439312  4422 tablet_bootstrap.cc:654] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.439285  4426 consensus_queue.cc:260] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.439709  4423 consensus_queue.cc:260] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.440102  4426 ts_tablet_manager.cc:1398] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:10:52.440279  4423 ts_tablet_manager.cc:1398] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:10:52.444746  4426 tablet_bootstrap.cc:492] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:52.445987  4428 tablet_bootstrap.cc:492] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:52.446933  4426 tablet_bootstrap.cc:654] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.447746  4428 ts_tablet_manager.cc:1367] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.029s	user 0.007s	sys 0.000s
I20240501 08:10:52.448630  4428 raft_consensus.cc:348] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.448889  4428 raft_consensus.cc:374] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.449018  4428 raft_consensus.cc:729] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.449342  4428 consensus_queue.cc:260] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.450155  4428 ts_tablet_manager.cc:1398] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:52.452183  4424 tablet_bootstrap.cc:492] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:52.452471  4424 ts_tablet_manager.cc:1367] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.003s
I20240501 08:10:52.453102  4426 tablet_bootstrap.cc:492] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:52.453442  4424 raft_consensus.cc:348] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.453804  4424 raft_consensus.cc:374] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.453867  4424 raft_consensus.cc:729] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.454165  4424 consensus_queue.cc:260] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.454707  4426 ts_tablet_manager.cc:1367] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240501 08:10:52.454872  4424 ts_tablet_manager.cc:1398] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:10:52.455597  4426 raft_consensus.cc:348] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.455873  4426 raft_consensus.cc:374] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.455994  4426 raft_consensus.cc:729] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.456413  4422 tablet_bootstrap.cc:492] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:52.456341  4426 consensus_queue.cc:260] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.456571  4422 ts_tablet_manager.cc:1367] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.011s
I20240501 08:10:52.457230  4422 raft_consensus.cc:348] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.457440  4422 raft_consensus.cc:374] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.457527  4422 raft_consensus.cc:729] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.457823  4426 ts_tablet_manager.cc:1398] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:52.457803  4422 consensus_queue.cc:260] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.458940  4422 ts_tablet_manager.cc:1398] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:52.459082  4370 raft_consensus.cc:482] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:52.459244  4370 raft_consensus.cc:504] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.460036  4370 leader_election.cc:288] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:52.460899  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae945fb63414d88b8ec336244aeb55e" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:52.461057  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae945fb63414d88b8ec336244aeb55e" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:52.461140  3225 raft_consensus.cc:2454] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:52.461289  3750 raft_consensus.cc:2454] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:52.461592  3425 leader_election.cc:302] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:52.461879  4370 raft_consensus.cc:2781] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:52.461989  4370 raft_consensus.cc:482] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:52.462074  4370 raft_consensus.cc:3037] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.499977  4370 raft_consensus.cc:504] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.500686  4370 leader_election.cc:288] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:52.501188  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae945fb63414d88b8ec336244aeb55e" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:52.501294  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae945fb63414d88b8ec336244aeb55e" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:52.501444  3750 raft_consensus.cc:3037] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.501480  3225 raft_consensus.cc:3037] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.503474  3225 raft_consensus.cc:2454] T 1ae945fb63414d88b8ec336244aeb55e P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:52.503877  3425 leader_election.cc:302] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:52.504209  4370 raft_consensus.cc:2781] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:52.504362  4370 raft_consensus.cc:686] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [term 1 LEADER]: Becoming Leader. State: Replica: 97336c9b436644a798689a443b2f272e, State: Running, Role: LEADER
I20240501 08:10:52.504657  4370 consensus_queue.cc:237] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:52.507684  3081 catalog_manager.cc:5484] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97336c9b436644a798689a443b2f272e (127.2.248.131). New cstate: current_term: 1 leader_uuid: "97336c9b436644a798689a443b2f272e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:52.509260  4317 raft_consensus.cc:482] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:52.509433  4317 raft_consensus.cc:504] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.510105  4317 leader_election.cc:288] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:52.510564  3619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867e0435e12a4255a9aaa7d42dbe45a7" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:52.510751  3619 raft_consensus.cc:2454] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:52.510977  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867e0435e12a4255a9aaa7d42dbe45a7" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:52.511233  3358 raft_consensus.cc:2454] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:52.511289  3817 leader_election.cc:302] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:52.511775  4317 raft_consensus.cc:2781] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:52.511906  4317 raft_consensus.cc:482] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:52.512013  4317 raft_consensus.cc:3037] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.564452  4438 raft_consensus.cc:482] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:52.564613  4438 raft_consensus.cc:504] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.565243  4438 leader_election.cc:288] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:52.565618  3619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01562e1616fb4a44af73d56f6b75b8e6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:52.565747  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01562e1616fb4a44af73d56f6b75b8e6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:52.565847  3619 raft_consensus.cc:2454] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:52.565994  3358 raft_consensus.cc:2454] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:52.566231  3817 leader_election.cc:302] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:52.566458  4438 raft_consensus.cc:2781] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:52.566550  4438 raft_consensus.cc:482] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:52.566610  4438 raft_consensus.cc:3037] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.601011  3750 raft_consensus.cc:2454] T 1ae945fb63414d88b8ec336244aeb55e P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:52.601043  4317 raft_consensus.cc:504] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.602028  4317 leader_election.cc:288] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:52.602510  3619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867e0435e12a4255a9aaa7d42dbe45a7" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:52.602761  3619 raft_consensus.cc:3037] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.602872  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867e0435e12a4255a9aaa7d42dbe45a7" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:52.603111  3358 raft_consensus.cc:3037] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.604009  4438 raft_consensus.cc:504] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.604887  4438 leader_election.cc:288] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:52.605232  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01562e1616fb4a44af73d56f6b75b8e6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:52.605455  3357 raft_consensus.cc:3037] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.606122  3358 raft_consensus.cc:2454] T 867e0435e12a4255a9aaa7d42dbe45a7 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:52.606273  3619 raft_consensus.cc:2454] T 867e0435e12a4255a9aaa7d42dbe45a7 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:52.606626  3818 leader_election.cc:302] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:52.607049  4438 raft_consensus.cc:2781] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:52.607612  4438 raft_consensus.cc:686] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:52.607591  3619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01562e1616fb4a44af73d56f6b75b8e6" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:52.607784  3619 raft_consensus.cc:3037] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.607931  4438 consensus_queue.cc:237] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:52.608217  3357 raft_consensus.cc:2454] T 01562e1616fb4a44af73d56f6b75b8e6 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:52.608776  3818 leader_election.cc:302] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:52.609066  4317 raft_consensus.cc:2781] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:52.610275  4317 raft_consensus.cc:686] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:52.610728  3619 raft_consensus.cc:2454] T 01562e1616fb4a44af73d56f6b75b8e6 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:52.610641  4317 consensus_queue.cc:237] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:52.611542  3081 catalog_manager.cc:5484] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:52.616755  3081 catalog_manager.cc:5484] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } }
W20240501 08:10:52.968533  4442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:52.968830  4442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:52.976316  4442 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:10:53.048856  4317 consensus_queue.cc:1035] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.052963  4370 consensus_queue.cc:1035] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.053462  4370 consensus_queue.cc:1035] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.078083  4317 consensus_queue.cc:1035] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.111586  4317 consensus_queue.cc:1035] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.123143  4438 consensus_queue.cc:1035] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 318.53 ms
  time per row: 0.159265 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:10:54.392571  3081 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:54.402112  3468 tablet_service.cc:1467] Processing CreateTablet for tablet 46e45c482291420da0989bb628d3485d (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.402493  3731 tablet_service.cc:1467] Processing CreateTablet for tablet 46e45c482291420da0989bb628d3485d (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.402608  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e45c482291420da0989bb628d3485d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.402925  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e45c482291420da0989bb628d3485d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.403350  3600 tablet_service.cc:1467] Processing CreateTablet for tablet b29d7b5336644659a0c3f7beef22d5b9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.403720  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29d7b5336644659a0c3f7beef22d5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.408529  3862 tablet_service.cc:1467] Processing CreateTablet for tablet b29d7b5336644659a0c3f7beef22d5b9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.408922  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29d7b5336644659a0c3f7beef22d5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.408957  3205 tablet_service.cc:1467] Processing CreateTablet for tablet 46e45c482291420da0989bb628d3485d (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.409328  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e45c482291420da0989bb628d3485d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.413930  3337 tablet_service.cc:1467] Processing CreateTablet for tablet b29d7b5336644659a0c3f7beef22d5b9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.414474  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29d7b5336644659a0c3f7beef22d5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.415608  4510 tablet_bootstrap.cc:492] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:54.415961  3206 tablet_service.cc:1467] Processing CreateTablet for tablet 1a434e9552e249f9b09853770852058f (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.416368  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a434e9552e249f9b09853770852058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.419217  4510 tablet_bootstrap.cc:654] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.422477  3469 tablet_service.cc:1467] Processing CreateTablet for tablet 1a434e9552e249f9b09853770852058f (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.422983  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a434e9552e249f9b09853770852058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.424639  3730 tablet_service.cc:1467] Processing CreateTablet for tablet 1a434e9552e249f9b09853770852058f (DEFAULT_TABLE table=rebalance_test_table_6 [id=6780eb1074b142fab8fb802e703fb170]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.425151  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a434e9552e249f9b09853770852058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.432566  4512 tablet_bootstrap.cc:492] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:54.436550  4512 tablet_bootstrap.cc:654] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.438277  4514 tablet_bootstrap.cc:492] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:54.441840  4512 tablet_bootstrap.cc:492] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:54.442005  4512 ts_tablet_manager.cc:1367] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.002s
I20240501 08:10:54.442646  4512 raft_consensus.cc:348] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.442811  4512 raft_consensus.cc:374] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.443380  4509 tablet_bootstrap.cc:492] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:54.443873  4514 tablet_bootstrap.cc:654] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.446584  4509 tablet_bootstrap.cc:654] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.448029  4512 raft_consensus.cc:729] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.448608  4512 consensus_queue.cc:260] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.449409  4514 tablet_bootstrap.cc:492] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:54.449558  4514 ts_tablet_manager.cc:1367] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.003s
I20240501 08:10:54.450024  4516 tablet_bootstrap.cc:492] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:54.450182  4514 raft_consensus.cc:348] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.450368  4514 raft_consensus.cc:374] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.450461  4514 raft_consensus.cc:729] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.450673  4514 consensus_queue.cc:260] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.451136  4514 ts_tablet_manager.cc:1398] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:10:54.452071  4512 ts_tablet_manager.cc:1398] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240501 08:10:54.454571  4516 tablet_bootstrap.cc:654] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.455472  4512 tablet_bootstrap.cc:492] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:54.457979  4509 tablet_bootstrap.cc:492] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:54.458128  4509 ts_tablet_manager.cc:1367] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.000s
I20240501 08:10:54.458703  4509 raft_consensus.cc:348] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.458864  4509 raft_consensus.cc:374] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.459971  4512 tablet_bootstrap.cc:654] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.460743  4511 tablet_bootstrap.cc:492] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:54.460942  4509 raft_consensus.cc:729] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.461901  4509 consensus_queue.cc:260] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.462716  4509 ts_tablet_manager.cc:1398] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:10:54.463047  4509 tablet_bootstrap.cc:492] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:54.464596  4509 tablet_bootstrap.cc:654] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.467651  4511 tablet_bootstrap.cc:654] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.467694  4510 tablet_bootstrap.cc:492] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:54.467836  4510 ts_tablet_manager.cc:1367] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.052s	user 0.021s	sys 0.014s
I20240501 08:10:54.468698  4510 raft_consensus.cc:348] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.468923  4510 raft_consensus.cc:374] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.469017  4510 raft_consensus.cc:729] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.469277  4510 consensus_queue.cc:260] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.469702  4509 tablet_bootstrap.cc:492] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:54.469859  4509 ts_tablet_manager.cc:1367] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240501 08:10:54.469908  4510 ts_tablet_manager.cc:1398] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:54.470559  4509 raft_consensus.cc:348] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.470775  4509 raft_consensus.cc:374] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.470861  4509 raft_consensus.cc:729] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.471100  4509 consensus_queue.cc:260] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.472240  4516 tablet_bootstrap.cc:492] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:54.472417  4516 ts_tablet_manager.cc:1367] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.000s
I20240501 08:10:54.473052  4509 ts_tablet_manager.cc:1398] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:10:54.473196  4512 tablet_bootstrap.cc:492] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:54.473412  4512 ts_tablet_manager.cc:1367] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.000s
I20240501 08:10:54.473769  4516 raft_consensus.cc:348] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.473973  4516 raft_consensus.cc:374] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.474058  4516 raft_consensus.cc:729] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.474396  4512 raft_consensus.cc:348] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.474649  4512 raft_consensus.cc:374] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.474777  4512 raft_consensus.cc:729] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.475018  4512 consensus_queue.cc:260] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.475564  4512 ts_tablet_manager.cc:1398] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:10:54.476745  4511 tablet_bootstrap.cc:492] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:54.476902  4516 consensus_queue.cc:260] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.477666  4516 ts_tablet_manager.cc:1398] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:10:54.477855  4511 ts_tablet_manager.cc:1367] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.000s
I20240501 08:10:54.478638  4511 raft_consensus.cc:348] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.478847  4511 raft_consensus.cc:374] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.478940  4511 raft_consensus.cc:729] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.479197  4511 consensus_queue.cc:260] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.479854  4511 ts_tablet_manager.cc:1398] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:54.480175  4511 tablet_bootstrap.cc:492] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:54.481595  4511 tablet_bootstrap.cc:654] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:54.485016  4511 tablet_bootstrap.cc:492] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:54.485175  4511 ts_tablet_manager.cc:1367] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:10:54.485759  4511 raft_consensus.cc:348] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.485970  4511 raft_consensus.cc:374] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:54.486052  4511 raft_consensus.cc:729] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:54.486315  4511 consensus_queue.cc:260] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.486829  4511 ts_tablet_manager.cc:1398] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:54.506443  4370 raft_consensus.cc:482] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:54.506665  4370 raft_consensus.cc:504] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.507344  4370 leader_election.cc:288] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:10:54.507814  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a434e9552e249f9b09853770852058f" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:54.507850  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a434e9552e249f9b09853770852058f" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:54.508021  3226 raft_consensus.cc:2454] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:54.508064  3750 raft_consensus.cc:2454] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:54.508509  3425 leader_election.cc:302] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:54.508742  4370 raft_consensus.cc:2781] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:54.508823  4370 raft_consensus.cc:482] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:54.508913  4370 raft_consensus.cc:3037] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.510759  4370 raft_consensus.cc:504] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.511260  4370 leader_election.cc:288] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Requested vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:10:54.511734  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a434e9552e249f9b09853770852058f" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:54.511766  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a434e9552e249f9b09853770852058f" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:54.511924  3226 raft_consensus.cc:3037] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.511935  3750 raft_consensus.cc:3037] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.513757  3226 raft_consensus.cc:2454] T 1a434e9552e249f9b09853770852058f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:54.513828  3750 raft_consensus.cc:2454] T 1a434e9552e249f9b09853770852058f P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:54.514134  3425 leader_election.cc:302] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:54.514375  4370 raft_consensus.cc:2781] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:54.514794  4370 raft_consensus.cc:686] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [term 1 LEADER]: Becoming Leader. State: Replica: 97336c9b436644a798689a443b2f272e, State: Running, Role: LEADER
I20240501 08:10:54.515364  4370 consensus_queue.cc:237] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:54.518242  3084 catalog_manager.cc:5484] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97336c9b436644a798689a443b2f272e (127.2.248.131). New cstate: current_term: 1 leader_uuid: "97336c9b436644a798689a443b2f272e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:54.548151  4302 raft_consensus.cc:482] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:54.548368  4302 raft_consensus.cc:504] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.549028  4302 leader_election.cc:288] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:54.549561  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e45c482291420da0989bb628d3485d" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:54.549556  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e45c482291420da0989bb628d3485d" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:54.549811  3226 raft_consensus.cc:2454] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:54.549818  3489 raft_consensus.cc:2454] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:54.550213  3687 leader_election.cc:302] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:54.550501  4302 raft_consensus.cc:2781] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:54.550608  4302 raft_consensus.cc:482] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:54.550696  4302 raft_consensus.cc:3037] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.552640  4302 raft_consensus.cc:504] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.553236  4302 leader_election.cc:288] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Requested vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:54.553681  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e45c482291420da0989bb628d3485d" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:54.553908  3489 raft_consensus.cc:3037] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.553853  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e45c482291420da0989bb628d3485d" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:54.554068  3226 raft_consensus.cc:3037] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.556746  3226 raft_consensus.cc:2454] T 46e45c482291420da0989bb628d3485d P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:54.556746  3489 raft_consensus.cc:2454] T 46e45c482291420da0989bb628d3485d P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:54.557111  3686 leader_election.cc:302] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:54.557425  4302 raft_consensus.cc:2781] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:54.557758  4302 raft_consensus.cc:686] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [term 1 LEADER]: Becoming Leader. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Running, Role: LEADER
I20240501 08:10:54.558017  4302 consensus_queue.cc:237] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:54.560741  3084 catalog_manager.cc:5484] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 reported cstate change: term changed from 0 to 1, leader changed from <none> to b526e10277944ab0b4744f9de18a8771 (127.2.248.133). New cstate: current_term: 1 leader_uuid: "b526e10277944ab0b4744f9de18a8771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:54.577154  4504 raft_consensus.cc:482] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:54.577342  4504 raft_consensus.cc:504] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.577993  4504 leader_election.cc:288] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:54.578406  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29d7b5336644659a0c3f7beef22d5b9" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:54.578490  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29d7b5336644659a0c3f7beef22d5b9" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:54.578600  3358 raft_consensus.cc:2454] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:54.578701  3620 raft_consensus.cc:2454] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:54.579007  3818 leader_election.cc:302] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:54.579244  4504 raft_consensus.cc:2781] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:54.579351  4504 raft_consensus.cc:482] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:54.579425  4504 raft_consensus.cc:3037] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.581295  4504 raft_consensus.cc:504] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.581821  4504 leader_election.cc:288] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:54.582234  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29d7b5336644659a0c3f7beef22d5b9" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:54.582275  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29d7b5336644659a0c3f7beef22d5b9" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:54.582422  3358 raft_consensus.cc:3037] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.582438  3620 raft_consensus.cc:3037] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.584663  3620 raft_consensus.cc:2454] T b29d7b5336644659a0c3f7beef22d5b9 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:54.584674  3358 raft_consensus.cc:2454] T b29d7b5336644659a0c3f7beef22d5b9 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:54.585013  3817 leader_election.cc:302] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:54.585268  4504 raft_consensus.cc:2781] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:54.585381  4504 raft_consensus.cc:686] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:54.585645  4504 consensus_queue.cc:237] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:54.588587  3084 catalog_manager.cc:5484] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:54.935329  4302 consensus_queue.cc:1035] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:54.941888  4302 consensus_queue.cc:1035] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:54.971099  4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:54.971747  4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:54.980762  4529 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:55.011242  4504 consensus_queue.cc:1035] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240501 08:10:55.022310  4504 consensus_queue.cc:1035] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:55.054997  4370 consensus_queue.cc:1035] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:55.075470  4370 consensus_queue.cc:1035] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 301.321 ms
  time per row: 0.150661 ms
I20240501 08:10:55.439530  3081 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:55.448395  3337 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf57c4a7fb443ab9527a779f91360cc (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.448762  3600 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf57c4a7fb443ab9527a779f91360cc (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.448904  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf57c4a7fb443ab9527a779f91360cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.449199  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf57c4a7fb443ab9527a779f91360cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.450168  3205 tablet_service.cc:1467] Processing CreateTablet for tablet aef6c6e24d184079a8c092f20e010f85 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.450460  3862 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf57c4a7fb443ab9527a779f91360cc (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.450584  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef6c6e24d184079a8c092f20e010f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.450832  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf57c4a7fb443ab9527a779f91360cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.452524  3338 tablet_service.cc:1467] Processing CreateTablet for tablet a42dd63dc2c747198b0e78bb3fa1969b (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.452857  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42dd63dc2c747198b0e78bb3fa1969b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.456339  3599 tablet_service.cc:1467] Processing CreateTablet for tablet a42dd63dc2c747198b0e78bb3fa1969b (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.456578  3469 tablet_service.cc:1467] Processing CreateTablet for tablet aef6c6e24d184079a8c092f20e010f85 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.456872  3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42dd63dc2c747198b0e78bb3fa1969b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.457121  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef6c6e24d184079a8c092f20e010f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.459837  3730 tablet_service.cc:1467] Processing CreateTablet for tablet aef6c6e24d184079a8c092f20e010f85 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.460364  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef6c6e24d184079a8c092f20e010f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.460418  4601 tablet_bootstrap.cc:492] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:55.462594  4601 tablet_bootstrap.cc:654] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.467612  3861 tablet_service.cc:1467] Processing CreateTablet for tablet a42dd63dc2c747198b0e78bb3fa1969b (DEFAULT_TABLE table=rebalance_test_table_7 [id=d54c5d42626d43ef93261c5f894e450a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:55.468277  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42dd63dc2c747198b0e78bb3fa1969b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:55.486821  4605 tablet_bootstrap.cc:492] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:55.488065  4606 tablet_bootstrap.cc:492] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:55.490372  4606 tablet_bootstrap.cc:654] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.490861  4605 tablet_bootstrap.cc:654] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.493777  4601 tablet_bootstrap.cc:492] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:55.494004  4601 ts_tablet_manager.cc:1367] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.034s	user 0.016s	sys 0.016s
I20240501 08:10:55.494112  4605 tablet_bootstrap.cc:492] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:55.494551  4605 ts_tablet_manager.cc:1367] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.002s
I20240501 08:10:55.494911  4601 raft_consensus.cc:348] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.495223  4601 raft_consensus.cc:374] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.495373  4601 raft_consensus.cc:729] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.495352  4605 raft_consensus.cc:348] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.495671  4605 raft_consensus.cc:374] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.495820  4605 raft_consensus.cc:729] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.495803  4601 consensus_queue.cc:260] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.496088  4605 consensus_queue.cc:260] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.496851  4605 ts_tablet_manager.cc:1398] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:10:55.497295  4601 ts_tablet_manager.cc:1398] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:10:55.497686  4605 tablet_bootstrap.cc:492] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:55.497777  4601 tablet_bootstrap.cc:492] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:55.499620  4605 tablet_bootstrap.cc:654] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.499646  4601 tablet_bootstrap.cc:654] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.499990  4606 tablet_bootstrap.cc:492] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:55.500253  4607 tablet_bootstrap.cc:492] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:55.501474  4606 ts_tablet_manager.cc:1367] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.000s
I20240501 08:10:55.502568  4604 tablet_bootstrap.cc:492] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:55.503983  4606 raft_consensus.cc:348] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.505312  4606 raft_consensus.cc:374] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.505625  4606 raft_consensus.cc:729] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.505903  4606 consensus_queue.cc:260] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.506477  4606 ts_tablet_manager.cc:1398] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.000s
I20240501 08:10:55.506757  4604 tablet_bootstrap.cc:654] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.508455  4607 tablet_bootstrap.cc:654] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.508810  4605 tablet_bootstrap.cc:492] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:55.508962  4605 ts_tablet_manager.cc:1367] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.002s
I20240501 08:10:55.509835  4605 raft_consensus.cc:348] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.510095  4605 raft_consensus.cc:374] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.510167  4605 raft_consensus.cc:729] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.510453  4605 consensus_queue.cc:260] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.511161  4605 ts_tablet_manager.cc:1398] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:10:55.512594  4602 tablet_bootstrap.cc:492] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:55.514145  4601 tablet_bootstrap.cc:492] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:55.514333  4601 ts_tablet_manager.cc:1367] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.003s
I20240501 08:10:55.514379  4602 tablet_bootstrap.cc:654] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.515144  4601 raft_consensus.cc:348] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.515416  4601 raft_consensus.cc:374] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.515530  4601 raft_consensus.cc:729] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.515784  4601 consensus_queue.cc:260] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.516343  4601 ts_tablet_manager.cc:1398] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240501 08:10:55.515853  4604 tablet_bootstrap.cc:492] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:55.516813  4607 tablet_bootstrap.cc:492] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:55.516815  4604 ts_tablet_manager.cc:1367] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.020s	user 0.000s	sys 0.005s
I20240501 08:10:55.517009  4607 ts_tablet_manager.cc:1367] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.000s
I20240501 08:10:55.517627  4604 raft_consensus.cc:348] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.517906  4604 raft_consensus.cc:374] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.517814  4607 raft_consensus.cc:348] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.518007  4607 raft_consensus.cc:374] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.518069  4604 raft_consensus.cc:729] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.518108  4607 raft_consensus.cc:729] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.518342  4607 consensus_queue.cc:260] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.518420  4604 consensus_queue.cc:260] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.518966  4607 ts_tablet_manager.cc:1398] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:10:55.519919  4604 ts_tablet_manager.cc:1398] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:10:55.520264  4604 tablet_bootstrap.cc:492] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:55.520344  4602 tablet_bootstrap.cc:492] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:55.520567  4602 ts_tablet_manager.cc:1367] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.004s
I20240501 08:10:55.521924  4604 tablet_bootstrap.cc:654] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.522851  4602 raft_consensus.cc:348] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.523113  4602 raft_consensus.cc:374] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.523257  4602 raft_consensus.cc:729] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.523792  4602 consensus_queue.cc:260] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.525033  4604 tablet_bootstrap.cc:492] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:55.525269  4604 ts_tablet_manager.cc:1367] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.001s
I20240501 08:10:55.525547  4602 ts_tablet_manager.cc:1398] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240501 08:10:55.526072  4604 raft_consensus.cc:348] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.526249  4604 raft_consensus.cc:374] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.526326  4604 raft_consensus.cc:729] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.526557  4604 consensus_queue.cc:260] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.527217  4604 ts_tablet_manager.cc:1398] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:10:55.656453  4565 raft_consensus.cc:482] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:55.656638  4565 raft_consensus.cc:504] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.657343  4565 leader_election.cc:288] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:55.657949  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef6c6e24d184079a8c092f20e010f85" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:55.657974  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef6c6e24d184079a8c092f20e010f85" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:55.658210  3225 raft_consensus.cc:2454] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:55.658207  3750 raft_consensus.cc:2454] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:10:55.658790  3425 leader_election.cc:302] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:55.659083  4565 raft_consensus.cc:2781] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:55.659188  4565 raft_consensus.cc:482] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:55.659276  4565 raft_consensus.cc:3037] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.661597  4565 raft_consensus.cc:504] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.662206  4565 leader_election.cc:288] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:55.662622  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef6c6e24d184079a8c092f20e010f85" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:55.662617  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef6c6e24d184079a8c092f20e010f85" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:55.662835  3225 raft_consensus.cc:3037] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.662832  3750 raft_consensus.cc:3037] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.665550  3225 raft_consensus.cc:2454] T aef6c6e24d184079a8c092f20e010f85 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:55.666044  3425 leader_election.cc:302] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:55.666307  4565 raft_consensus.cc:2781] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:55.666438  4565 raft_consensus.cc:686] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [term 1 LEADER]: Becoming Leader. State: Replica: 97336c9b436644a798689a443b2f272e, State: Running, Role: LEADER
I20240501 08:10:55.666935  4565 consensus_queue.cc:237] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:55.667088  3750 raft_consensus.cc:2454] T aef6c6e24d184079a8c092f20e010f85 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:10:55.670437  3082 catalog_manager.cc:5484] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97336c9b436644a798689a443b2f272e (127.2.248.131). New cstate: current_term: 1 leader_uuid: "97336c9b436644a798689a443b2f272e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:55.681087  3971 raft_consensus.cc:482] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:55.681309  3971 raft_consensus.cc:504] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.682011  3971 leader_election.cc:288] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:10:55.682499  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42dd63dc2c747198b0e78bb3fa1969b" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:55.682741  3357 raft_consensus.cc:2454] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:55.683152  3556 leader_election.cc:302] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:55.683457  3971 raft_consensus.cc:2781] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:55.683563  3971 raft_consensus.cc:482] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:55.683651  3971 raft_consensus.cc:3037] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.683559  3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42dd63dc2c747198b0e78bb3fa1969b" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0" is_pre_election: true
I20240501 08:10:55.683804  3881 raft_consensus.cc:2454] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:55.685706  3971 raft_consensus.cc:504] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.686246  3971 leader_election.cc:288] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:10:55.686687  3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42dd63dc2c747198b0e78bb3fa1969b" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0"
I20240501 08:10:55.686710  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42dd63dc2c747198b0e78bb3fa1969b" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:55.686918  3881 raft_consensus.cc:3037] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.686931  3357 raft_consensus.cc:3037] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.689677  3357 raft_consensus.cc:2454] T a42dd63dc2c747198b0e78bb3fa1969b P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:55.689677  3881 raft_consensus.cc:2454] T a42dd63dc2c747198b0e78bb3fa1969b P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:55.690030  3556 leader_election.cc:302] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:55.690315  3971 raft_consensus.cc:2781] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:55.690753  3971 raft_consensus.cc:686] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [term 1 LEADER]: Becoming Leader. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Running, Role: LEADER
I20240501 08:10:55.691082  3971 consensus_queue.cc:237] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:55.694631  3082 catalog_manager.cc:5484] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132). New cstate: current_term: 1 leader_uuid: "29eea43c04ae49edb64ba8803bf09b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:55.794525  4504 raft_consensus.cc:482] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:55.794735  4504 raft_consensus.cc:504] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.795362  4504 leader_election.cc:288] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:55.795929  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf57c4a7fb443ab9527a779f91360cc" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:55.796104  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf57c4a7fb443ab9527a779f91360cc" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:55.796217  3357 raft_consensus.cc:2454] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:55.796391  3620 raft_consensus.cc:2454] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:10:55.796794  3818 leader_election.cc:302] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:55.797062  4504 raft_consensus.cc:2781] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:55.797152  4504 raft_consensus.cc:482] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:55.797256  4504 raft_consensus.cc:3037] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.799037  4504 raft_consensus.cc:504] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.800066  3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf57c4a7fb443ab9527a779f91360cc" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:55.800280  3357 raft_consensus.cc:3037] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.800225  3619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf57c4a7fb443ab9527a779f91360cc" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:55.800415  3619 raft_consensus.cc:3037] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.803092  3619 raft_consensus.cc:2454] T 6bf57c4a7fb443ab9527a779f91360cc P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:55.803120  3357 raft_consensus.cc:2454] T 6bf57c4a7fb443ab9527a779f91360cc P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:10:55.803366  4504 leader_election.cc:288] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:10:55.803555  3818 leader_election.cc:302] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:55.803823  4504 raft_consensus.cc:2781] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:55.803959  4504 raft_consensus.cc:686] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:10:55.804260  4504 consensus_queue.cc:237] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:55.807205  3082 catalog_manager.cc:5484] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:56.072630  3971 consensus_queue.cc:1035] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:56.079493  4617 consensus_queue.cc:1035] T a42dd63dc2c747198b0e78bb3fa1969b P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:56.138795  4565 consensus_queue.cc:1035] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:56.147046  4565 consensus_queue.cc:1035] T aef6c6e24d184079a8c092f20e010f85 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:56.192345  4620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:56.192694  4620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:56.200871  4620 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:56.207648  4504 consensus_queue.cc:1035] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:56.213368  4585 consensus_queue.cc:1035] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'

INSERT report
    rows total: 2000
    time total: 294.19 ms
  time per row: 0.147095 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:10:57.582185  3085 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:57.590831  3730 tablet_service.cc:1467] Processing CreateTablet for tablet 1002d7b484654f908eb6732dabb63e4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.591363  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1002d7b484654f908eb6732dabb63e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.591753  3205 tablet_service.cc:1467] Processing CreateTablet for tablet 1002d7b484654f908eb6732dabb63e4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.591934  3862 tablet_service.cc:1467] Processing CreateTablet for tablet bbda0193ab414fdeaa1748c23e3c889c (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.591972  3469 tablet_service.cc:1467] Processing CreateTablet for tablet 1002d7b484654f908eb6732dabb63e4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.592180  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1002d7b484654f908eb6732dabb63e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.592355  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbda0193ab414fdeaa1748c23e3c889c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.592407  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1002d7b484654f908eb6732dabb63e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.595927  3599 tablet_service.cc:1467] Processing CreateTablet for tablet bbda0193ab414fdeaa1748c23e3c889c (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.599213  3338 tablet_service.cc:1467] Processing CreateTablet for tablet bbda0193ab414fdeaa1748c23e3c889c (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.599380  3731 tablet_service.cc:1467] Processing CreateTablet for tablet edb2ba21e0df4769b9ca97b451c36d1b (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.599745  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbda0193ab414fdeaa1748c23e3c889c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.599881  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb2ba21e0df4769b9ca97b451c36d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.601763  4696 tablet_bootstrap.cc:492] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:57.603416  4696 tablet_bootstrap.cc:654] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.604676  3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbda0193ab414fdeaa1748c23e3c889c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.610571  3206 tablet_service.cc:1467] Processing CreateTablet for tablet edb2ba21e0df4769b9ca97b451c36d1b (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.611084  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb2ba21e0df4769b9ca97b451c36d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.612538  4699 tablet_bootstrap.cc:492] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:57.612813  3468 tablet_service.cc:1467] Processing CreateTablet for tablet edb2ba21e0df4769b9ca97b451c36d1b (DEFAULT_TABLE table=rebalance_test_table_8 [id=8551492375a849069854e0d6a25e175a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:57.613361  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb2ba21e0df4769b9ca97b451c36d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.614168  4699 tablet_bootstrap.cc:654] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.615039  4700 tablet_bootstrap.cc:492] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:57.617874  4700 tablet_bootstrap.cc:654] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.619722  4699 tablet_bootstrap.cc:492] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:57.621073  4696 tablet_bootstrap.cc:492] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:57.621299  4696 ts_tablet_manager.cc:1367] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.008s
I20240501 08:10:57.620369  4699 ts_tablet_manager.cc:1367] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240501 08:10:57.622133  4699 raft_consensus.cc:348] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.622349  4699 raft_consensus.cc:374] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.622437  4699 raft_consensus.cc:729] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.622663  4699 consensus_queue.cc:260] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.622713  4696 raft_consensus.cc:348] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.623054  4696 raft_consensus.cc:374] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.623226  4696 raft_consensus.cc:729] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.623558  4696 consensus_queue.cc:260] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.625666  4696 ts_tablet_manager.cc:1398] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 08:10:57.626077  4699 ts_tablet_manager.cc:1398] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.000s
I20240501 08:10:57.626811  4699 tablet_bootstrap.cc:492] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:57.628422  4699 tablet_bootstrap.cc:654] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.634217  4704 tablet_bootstrap.cc:492] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:57.637115  4704 tablet_bootstrap.cc:654] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.647208  4706 tablet_bootstrap.cc:492] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:57.650506  4706 tablet_bootstrap.cc:654] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.653941  4696 tablet_bootstrap.cc:492] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:57.656534  4696 tablet_bootstrap.cc:654] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.660218  4706 tablet_bootstrap.cc:492] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:57.661355  4706 ts_tablet_manager.cc:1367] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.000s
I20240501 08:10:57.662593  4706 raft_consensus.cc:348] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.663798  4700 tablet_bootstrap.cc:492] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:57.663980  4700 ts_tablet_manager.cc:1367] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.049s	user 0.016s	sys 0.025s
I20240501 08:10:57.664754  4706 raft_consensus.cc:374] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.664871  4706 raft_consensus.cc:729] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.664852  4700 raft_consensus.cc:348] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.665047  4700 raft_consensus.cc:374] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.665143  4700 raft_consensus.cc:729] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.665092  4706 consensus_queue.cc:260] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.665359  4700 consensus_queue.cc:260] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.666025  4696 tablet_bootstrap.cc:492] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:57.666296  4696 ts_tablet_manager.cc:1367] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.012s	user 0.000s	sys 0.009s
I20240501 08:10:57.667338  4696 raft_consensus.cc:348] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.667726  4706 ts_tablet_manager.cc:1398] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 08:10:57.668877  4700 ts_tablet_manager.cc:1398] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240501 08:10:57.669186  4699 tablet_bootstrap.cc:492] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:57.669401  4699 ts_tablet_manager.cc:1367] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.043s	user 0.004s	sys 0.012s
I20240501 08:10:57.670190  4699 raft_consensus.cc:348] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.670799  4699 raft_consensus.cc:374] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.671020  4699 raft_consensus.cc:729] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.671979  4697 tablet_bootstrap.cc:492] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:57.671475  4699 consensus_queue.cc:260] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.673718  4699 ts_tablet_manager.cc:1398] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 08:10:57.675766  4697 tablet_bootstrap.cc:654] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.667711  4696 raft_consensus.cc:374] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.676774  4704 tablet_bootstrap.cc:492] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:57.676913  4696 raft_consensus.cc:729] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.677284  4696 consensus_queue.cc:260] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.678197  4696 ts_tablet_manager.cc:1398] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.002s
I20240501 08:10:57.678807  4704 ts_tablet_manager.cc:1367] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.045s	user 0.005s	sys 0.025s
I20240501 08:10:57.679819  4704 raft_consensus.cc:348] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.680276  4704 raft_consensus.cc:374] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.680397  4704 raft_consensus.cc:729] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.680636  4704 consensus_queue.cc:260] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.681321  4704 ts_tablet_manager.cc:1398] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:57.682387  4697 tablet_bootstrap.cc:492] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:57.682564  4697 ts_tablet_manager.cc:1367] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.002s
I20240501 08:10:57.683321  4697 raft_consensus.cc:348] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.683476  4697 raft_consensus.cc:374] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.683562  4697 raft_consensus.cc:729] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.683750  4697 consensus_queue.cc:260] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.684247  4697 ts_tablet_manager.cc:1398] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:10:57.684561  4697 tablet_bootstrap.cc:492] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:57.686115  4697 tablet_bootstrap.cc:654] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.687832  4697 tablet_bootstrap.cc:492] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:57.687985  4697 ts_tablet_manager.cc:1367] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:10:57.688715  4697 raft_consensus.cc:348] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.688886  4697 raft_consensus.cc:374] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.688966  4697 raft_consensus.cc:729] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.689177  4697 consensus_queue.cc:260] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.689616  4697 ts_tablet_manager.cc:1398] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:10:57.730698  4526 raft_consensus.cc:482] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:57.730880  4526 raft_consensus.cc:504] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.731580  4526 leader_election.cc:288] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:57.732004  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7b484654f908eb6732dabb63e4f" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:57.732012  3488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7b484654f908eb6732dabb63e4f" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:57.732252  3225 raft_consensus.cc:2454] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:57.732276  3488 raft_consensus.cc:2454] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:57.732769  3687 leader_election.cc:302] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:57.733052  4526 raft_consensus.cc:2781] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:57.733168  4526 raft_consensus.cc:482] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:57.733247  4526 raft_consensus.cc:3037] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.735342  4526 raft_consensus.cc:504] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.735983  4526 leader_election.cc:288] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Requested vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:57.736428  3488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7b484654f908eb6732dabb63e4f" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:57.736402  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7b484654f908eb6732dabb63e4f" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:57.736608  3488 raft_consensus.cc:3037] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.736613  3225 raft_consensus.cc:3037] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.739395  3971 raft_consensus.cc:482] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:57.739411  3488 raft_consensus.cc:2454] T 1002d7b484654f908eb6732dabb63e4f P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:57.739414  3225 raft_consensus.cc:2454] T 1002d7b484654f908eb6732dabb63e4f P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:10:57.739538  3971 raft_consensus.cc:504] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.739795  3686 leader_election.cc:302] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:57.740069  4526 raft_consensus.cc:2781] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:57.740209  3971 leader_election.cc:288] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:57.740535  4526 raft_consensus.cc:686] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [term 1 LEADER]: Becoming Leader. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Running, Role: LEADER
I20240501 08:10:57.740716  3356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbda0193ab414fdeaa1748c23e3c889c" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:57.740931  3356 raft_consensus.cc:2454] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:57.740918  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbda0193ab414fdeaa1748c23e3c889c" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0" is_pre_election: true
I20240501 08:10:57.740892  4526 consensus_queue.cc:237] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:10:57.741111  3882 raft_consensus.cc:2454] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:57.741369  3556 leader_election.cc:302] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:57.741640  3971 raft_consensus.cc:2781] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:57.741752  3971 raft_consensus.cc:482] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:57.741843  3971 raft_consensus.cc:3037] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.744084  3971 raft_consensus.cc:504] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.744109  3076 catalog_manager.cc:5484] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 reported cstate change: term changed from 0 to 1, leader changed from <none> to b526e10277944ab0b4744f9de18a8771 (127.2.248.133). New cstate: current_term: 1 leader_uuid: "b526e10277944ab0b4744f9de18a8771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:57.744711  3971 leader_election.cc:288] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Requested vote from peers 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:57.745206  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbda0193ab414fdeaa1748c23e3c889c" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0"
I20240501 08:10:57.745251  3356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbda0193ab414fdeaa1748c23e3c889c" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:57.745404  3882 raft_consensus.cc:3037] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.745455  3356 raft_consensus.cc:3037] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.748232  3356 raft_consensus.cc:2454] T bbda0193ab414fdeaa1748c23e3c889c P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:57.748239  3882 raft_consensus.cc:2454] T bbda0193ab414fdeaa1748c23e3c889c P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:57.748603  3556 leader_election.cc:302] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:57.748890  3971 raft_consensus.cc:2781] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:57.749244  3971 raft_consensus.cc:686] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [term 1 LEADER]: Becoming Leader. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Running, Role: LEADER
I20240501 08:10:57.749518  3971 consensus_queue.cc:237] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:10:57.752197  3076 catalog_manager.cc:5484] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132). New cstate: current_term: 1 leader_uuid: "29eea43c04ae49edb64ba8803bf09b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:57.843127  4526 raft_consensus.cc:482] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:57.843291  4365 raft_consensus.cc:482] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:57.843374  4526 raft_consensus.cc:504] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.843513  4365 raft_consensus.cc:504] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.844154  4365 leader_election.cc:288] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:57.844167  4526 leader_election.cc:288] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:57.844779  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:10:57.844812  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:57.844848  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:57.844977  3751 raft_consensus.cc:2454] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:57.844995  3225 raft_consensus.cc:2454] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:57.845050  3489 raft_consensus.cc:2454] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:10:57.845459  3162 leader_election.cc:302] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:57.845466  3686 leader_election.cc:302] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:10:57.844815  3488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:57.845654  3488 raft_consensus.cc:2454] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:57.845762  4526 raft_consensus.cc:2781] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:57.845860  4526 raft_consensus.cc:482] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:57.845896  4365 raft_consensus.cc:2781] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:57.845961  4526 raft_consensus.cc:3037] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.846011  4365 raft_consensus.cc:482] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:57.846122  4365 raft_consensus.cc:3037] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.848040  4365 raft_consensus.cc:504] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.848578  4365 leader_election.cc:288] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:10:57.848965  3488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:57.848944  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:57.849153  3488 raft_consensus.cc:3037] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.851137  3488 raft_consensus.cc:2454] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:10:57.851533  3161 leader_election.cc:302] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:57.851814  4365 raft_consensus.cc:2781] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:57.852218  4365 raft_consensus.cc:686] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 LEADER]: Becoming Leader. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Running, Role: LEADER
I20240501 08:10:57.852537  4365 consensus_queue.cc:237] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.855329  3076 catalog_manager.cc:5484] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:57.861811  4526 raft_consensus.cc:504] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:10:57.862246  3751 raft_consensus.cc:2379] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in current term 1: Already voted for candidate b526e10277944ab0b4744f9de18a8771 in this term.
I20240501 08:10:57.862502  4526 leader_election.cc:288] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Requested vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925)
I20240501 08:10:57.863312  3488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:57.863638  3488 raft_consensus.cc:2379] T edb2ba21e0df4769b9ca97b451c36d1b P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b526e10277944ab0b4744f9de18a8771 in current term 1: Already voted for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in this term.
I20240501 08:10:57.863603  3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2ba21e0df4769b9ca97b451c36d1b" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:10:57.864280  3687 leader_election.cc:302] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b526e10277944ab0b4744f9de18a8771; no voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5
I20240501 08:10:57.864537  4526 raft_consensus.cc:2726] T edb2ba21e0df4769b9ca97b451c36d1b P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240501 08:10:58.241689  4717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:58.241992  4717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:58.249581  4717 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:58.280550  4365 consensus_queue.cc:1035] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:58.285981  4365 consensus_queue.cc:1035] T edb2ba21e0df4769b9ca97b451c36d1b P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:58.293891  3971 consensus_queue.cc:1035] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240501 08:10:58.306435  4526 consensus_queue.cc:1035] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:58.340443  3971 consensus_queue.cc:1035] T bbda0193ab414fdeaa1748c23e3c889c P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:58.341192  4740 consensus_queue.cc:1035] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 250.923 ms
  time per row: 0.125462 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:10:59.688350  3076 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:10:59.697090  3862 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7c327c7b794abca7cda496441e2ae6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.697551  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7c327c7b794abca7cda496441e2ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.697551  3599 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7c327c7b794abca7cda496441e2ae6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.697911  3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7c327c7b794abca7cda496441e2ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.698374  3338 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7c327c7b794abca7cda496441e2ae6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.698679  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7c327c7b794abca7cda496441e2ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.703318  3600 tablet_service.cc:1467] Processing CreateTablet for tablet c575114ae3b04353817dd802b07d5011 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.703658  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c575114ae3b04353817dd802b07d5011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.703905  3861 tablet_service.cc:1467] Processing CreateTablet for tablet c575114ae3b04353817dd802b07d5011 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.704241  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c575114ae3b04353817dd802b07d5011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.704237  3468 tablet_service.cc:1467] Processing CreateTablet for tablet b8ac230309f0427dab6698aba1fb1e55 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.704607  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ac230309f0427dab6698aba1fb1e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.704582  3205 tablet_service.cc:1467] Processing CreateTablet for tablet b8ac230309f0427dab6698aba1fb1e55 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.704975  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ac230309f0427dab6698aba1fb1e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.708475  3337 tablet_service.cc:1467] Processing CreateTablet for tablet c575114ae3b04353817dd802b07d5011 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.708818  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c575114ae3b04353817dd802b07d5011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.709383  3731 tablet_service.cc:1467] Processing CreateTablet for tablet b8ac230309f0427dab6698aba1fb1e55 (DEFAULT_TABLE table=rebalance_test_table_9 [id=aa17477b2b8d48c79a1f0abebdd4ee5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.709784  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ac230309f0427dab6698aba1fb1e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.714277  4798 tablet_bootstrap.cc:492] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:59.717788  4798 tablet_bootstrap.cc:654] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.717804  4797 tablet_bootstrap.cc:492] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:59.722184  4797 tablet_bootstrap.cc:654] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.722337  4801 tablet_bootstrap.cc:492] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:59.722821  4800 tablet_bootstrap.cc:492] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:10:59.724020  4801 tablet_bootstrap.cc:654] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.725706  4799 tablet_bootstrap.cc:492] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:10:59.725656  4800 tablet_bootstrap.cc:654] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.727365  4799 tablet_bootstrap.cc:654] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.727941  4797 tablet_bootstrap.cc:492] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:59.728111  4797 ts_tablet_manager.cc:1367] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240501 08:10:59.729111  4797 raft_consensus.cc:348] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.729501  4797 raft_consensus.cc:374] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.729658  4797 raft_consensus.cc:729] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.729997  4797 consensus_queue.cc:260] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.730697  4797 ts_tablet_manager.cc:1398] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:10:59.731077  4797 tablet_bootstrap.cc:492] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:10:59.732015  4800 tablet_bootstrap.cc:492] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:10:59.732247  4800 ts_tablet_manager.cc:1367] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240501 08:10:59.732620  4797 tablet_bootstrap.cc:654] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.732944  4800 raft_consensus.cc:348] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.733239  4800 raft_consensus.cc:374] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.733348  4800 raft_consensus.cc:729] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.733656  4800 consensus_queue.cc:260] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.736236  4800 ts_tablet_manager.cc:1398] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:10:59.750734  4808 tablet_bootstrap.cc:492] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:10:59.750788  4799 tablet_bootstrap.cc:492] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:10:59.751039  4799 ts_tablet_manager.cc:1367] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.005s
I20240501 08:10:59.752161  4799 raft_consensus.cc:348] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.752662  4799 raft_consensus.cc:374] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.752823  4799 raft_consensus.cc:729] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.753181  4798 tablet_bootstrap.cc:492] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:59.753182  4799 consensus_queue.cc:260] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.753319  4798 ts_tablet_manager.cc:1367] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.039s	user 0.006s	sys 0.016s
I20240501 08:10:59.754508  4801 tablet_bootstrap.cc:492] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:59.754755  4801 ts_tablet_manager.cc:1367] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.000s
I20240501 08:10:59.755793  4801 raft_consensus.cc:348] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.755614  4798 raft_consensus.cc:348] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.756214  4801 raft_consensus.cc:374] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.756168  4798 raft_consensus.cc:374] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.756433  4801 raft_consensus.cc:729] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.756496  4798 raft_consensus.cc:729] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.756888  4801 consensus_queue.cc:260] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.757138  4798 consensus_queue.cc:260] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.757730  4808 tablet_bootstrap.cc:654] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.757863  4801 ts_tablet_manager.cc:1398] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:10:59.758275  4801 tablet_bootstrap.cc:492] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:10:59.759925  4798 ts_tablet_manager.cc:1398] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240501 08:10:59.760409  4798 tablet_bootstrap.cc:492] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:10:59.761415  4799 ts_tablet_manager.cc:1398] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240501 08:10:59.762362  4801 tablet_bootstrap.cc:654] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.763286  4798 tablet_bootstrap.cc:654] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.765470  4798 tablet_bootstrap.cc:492] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:10:59.765642  4798 ts_tablet_manager.cc:1367] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.002s
I20240501 08:10:59.770656  4798 raft_consensus.cc:348] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.770886  4801 tablet_bootstrap.cc:492] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:10:59.770992  4798 raft_consensus.cc:374] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.771059  4801 ts_tablet_manager.cc:1367] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.008s
I20240501 08:10:59.771193  4798 raft_consensus.cc:729] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.771484  4798 consensus_queue.cc:260] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.771818  4801 raft_consensus.cc:348] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.772039  4801 raft_consensus.cc:374] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.772140  4801 raft_consensus.cc:729] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.772322  4798 ts_tablet_manager.cc:1398] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.002s
I20240501 08:10:59.772414  4801 consensus_queue.cc:260] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.774861  4801 ts_tablet_manager.cc:1398] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:10:59.784688  4797 tablet_bootstrap.cc:492] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:10:59.784880  4797 ts_tablet_manager.cc:1367] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.054s	user 0.014s	sys 0.015s
I20240501 08:10:59.785651  4797 raft_consensus.cc:348] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.785876  4797 raft_consensus.cc:374] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.785938  4797 raft_consensus.cc:729] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.786144  4797 consensus_queue.cc:260] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.786717  4797 ts_tablet_manager.cc:1398] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:10:59.792493  4808 tablet_bootstrap.cc:492] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:10:59.792649  4808 ts_tablet_manager.cc:1367] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.042s	user 0.004s	sys 0.019s
I20240501 08:10:59.793231  4808 raft_consensus.cc:348] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.793386  4808 raft_consensus.cc:374] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.793480  4808 raft_consensus.cc:729] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.793689  4808 consensus_queue.cc:260] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.794157  4808 ts_tablet_manager.cc:1398] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:10:59.825771  4503 raft_consensus.cc:482] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:59.825949  4503 raft_consensus.cc:504] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.826610  4503 leader_election.cc:288] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:10:59.827100  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c575114ae3b04353817dd802b07d5011" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:10:59.827118  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c575114ae3b04353817dd802b07d5011" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0" is_pre_election: true
I20240501 08:10:59.827348  3882 raft_consensus.cc:2454] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 0.
I20240501 08:10:59.827360  3620 raft_consensus.cc:2454] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 0.
I20240501 08:10:59.827787  3291 leader_election.cc:302] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:10:59.828050  4503 raft_consensus.cc:2781] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:59.828176  4503 raft_consensus.cc:482] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:59.828266  4503 raft_consensus.cc:3037] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.830017  4503 raft_consensus.cc:504] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.830598  4503 leader_election.cc:288] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 election: Requested vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:10:59.831035  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c575114ae3b04353817dd802b07d5011" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:10:59.831264  3620 raft_consensus.cc:3037] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.831509  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c575114ae3b04353817dd802b07d5011" candidate_uuid: "1979c6dfa70f42699a57e8210a28b878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0"
I20240501 08:10:59.831709  3882 raft_consensus.cc:3037] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.834002  3620 raft_consensus.cc:2454] T c575114ae3b04353817dd802b07d5011 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 1.
I20240501 08:10:59.834288  3882 raft_consensus.cc:2454] T c575114ae3b04353817dd802b07d5011 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1979c6dfa70f42699a57e8210a28b878 in term 1.
I20240501 08:10:59.834491  3293 leader_election.cc:302] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:59.834810  4503 raft_consensus.cc:2781] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:59.835238  4503 raft_consensus.cc:686] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [term 1 LEADER]: Becoming Leader. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Running, Role: LEADER
I20240501 08:10:59.835955  4503 consensus_queue.cc:237] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:10:59.839068  3085 catalog_manager.cc:5484] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "1979c6dfa70f42699a57e8210a28b878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:59.840986  4735 raft_consensus.cc:482] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:59.841140  4735 raft_consensus.cc:504] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.841781  4735 leader_election.cc:288] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:59.842310  3356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c327c7b794abca7cda496441e2ae6" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:10:59.842330  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c327c7b794abca7cda496441e2ae6" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0" is_pre_election: true
I20240501 08:10:59.842535  3356 raft_consensus.cc:2454] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:59.842684  3882 raft_consensus.cc:2454] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:10:59.842974  3556 leader_election.cc:302] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:59.843281  4735 raft_consensus.cc:2781] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:59.843392  4735 raft_consensus.cc:482] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:59.843489  4735 raft_consensus.cc:3037] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.846464  4735 raft_consensus.cc:504] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.847124  4735 leader_election.cc:288] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Requested vote from peers 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:10:59.847513  3356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c327c7b794abca7cda496441e2ae6" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:10:59.847537  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7c327c7b794abca7cda496441e2ae6" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0"
I20240501 08:10:59.847735  3356 raft_consensus.cc:3037] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.847820  3882 raft_consensus.cc:3037] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.849882  3356 raft_consensus.cc:2454] T 6c7c327c7b794abca7cda496441e2ae6 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:59.850239  3556 leader_election.cc:302] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:10:59.850651  4735 raft_consensus.cc:2781] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:59.850966  3882 raft_consensus.cc:2454] T 6c7c327c7b794abca7cda496441e2ae6 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:10:59.852450  4735 raft_consensus.cc:686] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 LEADER]: Becoming Leader. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Running, Role: LEADER
I20240501 08:10:59.852861  4735 consensus_queue.cc:237] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } }
I20240501 08:10:59.855966  3085 catalog_manager.cc:5484] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132). New cstate: current_term: 1 leader_uuid: "29eea43c04ae49edb64ba8803bf09b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:59.962366  4715 raft_consensus.cc:482] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:59.962543  4715 raft_consensus.cc:504] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.963115  4715 leader_election.cc:288] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:10:59.963660  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ac230309f0427dab6698aba1fb1e55" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:10:59.963660  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ac230309f0427dab6698aba1fb1e55" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:10:59.963877  3751 raft_consensus.cc:2454] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:59.963876  3489 raft_consensus.cc:2454] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:10:59.964294  3161 leader_election.cc:302] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:59.964547  4715 raft_consensus.cc:2781] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:59.964653  4715 raft_consensus.cc:482] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:59.964741  4715 raft_consensus.cc:3037] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.966601  4715 raft_consensus.cc:504] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.967128  4715 leader_election.cc:288] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Requested vote from peers 97336c9b436644a798689a443b2f272e (127.2.248.131:44211), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:10:59.967614  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ac230309f0427dab6698aba1fb1e55" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:10:59.967626  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ac230309f0427dab6698aba1fb1e55" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:10:59.967806  3751 raft_consensus.cc:3037] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.967836  3489 raft_consensus.cc:3037] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.969758  3751 raft_consensus.cc:2454] T b8ac230309f0427dab6698aba1fb1e55 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:10:59.969769  3489 raft_consensus.cc:2454] T b8ac230309f0427dab6698aba1fb1e55 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:10:59.970132  3162 leader_election.cc:302] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:10:59.970459  4715 raft_consensus.cc:2781] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:59.970782  4715 raft_consensus.cc:686] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 LEADER]: Becoming Leader. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Running, Role: LEADER
I20240501 08:10:59.971047  4715 consensus_queue.cc:237] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:10:59.974172  3085 catalog_manager.cc:5484] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:00.235194  4503 consensus_queue.cc:1035] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:00.242486  4503 consensus_queue.cc:1035] T c575114ae3b04353817dd802b07d5011 P 1979c6dfa70f42699a57e8210a28b878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:00.271929  4735 consensus_queue.cc:1035] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:00.277204  4814 consensus_queue.cc:1035] T 6c7c327c7b794abca7cda496441e2ae6 P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:00.363895  4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:00.364279  4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:00.372452  4715 consensus_queue.cc:1035] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:00.373148  4818 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:00.380070  4715 consensus_queue.cc:1035] T b8ac230309f0427dab6698aba1fb1e55 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'

INSERT report
    rows total: 2000
    time total: 272.417 ms
  time per row: 0.136208 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:11:01.739156  3078 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_10"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:11:01.749236  3731 tablet_service.cc:1467] Processing CreateTablet for tablet 89cf58d92eda427d8c0e88d17a8e0c92 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.749378  3468 tablet_service.cc:1467] Processing CreateTablet for tablet 89cf58d92eda427d8c0e88d17a8e0c92 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.749732  3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89cf58d92eda427d8c0e88d17a8e0c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.749800  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89cf58d92eda427d8c0e88d17a8e0c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.750074  3206 tablet_service.cc:1467] Processing CreateTablet for tablet 988163c67d974e01b512a163da9790c3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.750488  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988163c67d974e01b512a163da9790c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.750454  3861 tablet_service.cc:1467] Processing CreateTablet for tablet e58f1eebd5da4a9b8d2ce592f24bc5d2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.750839  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58f1eebd5da4a9b8d2ce592f24bc5d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.752771  3205 tablet_service.cc:1467] Processing CreateTablet for tablet 89cf58d92eda427d8c0e88d17a8e0c92 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.753113  3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89cf58d92eda427d8c0e88d17a8e0c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.756644  3600 tablet_service.cc:1467] Processing CreateTablet for tablet e58f1eebd5da4a9b8d2ce592f24bc5d2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.756721  3338 tablet_service.cc:1467] Processing CreateTablet for tablet e58f1eebd5da4a9b8d2ce592f24bc5d2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.757125  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58f1eebd5da4a9b8d2ce592f24bc5d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.757161  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58f1eebd5da4a9b8d2ce592f24bc5d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.757524  3469 tablet_service.cc:1467] Processing CreateTablet for tablet 988163c67d974e01b512a163da9790c3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.757872  3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988163c67d974e01b512a163da9790c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.761237  3730 tablet_service.cc:1467] Processing CreateTablet for tablet 988163c67d974e01b512a163da9790c3 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3034b025091541bfb0a2407b28830ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.761648  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988163c67d974e01b512a163da9790c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.775902  4900 tablet_bootstrap.cc:492] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:11:01.776239  4897 tablet_bootstrap.cc:492] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:11:01.776078  4898 tablet_bootstrap.cc:492] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:11:01.777841  4900 tablet_bootstrap.cc:654] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.778398  4896 tablet_bootstrap.cc:492] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:11:01.779242  4897 tablet_bootstrap.cc:654] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.779824  4899 tablet_bootstrap.cc:492] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:11:01.779840  4898 tablet_bootstrap.cc:654] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.780951  4896 tablet_bootstrap.cc:654] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.785533  4901 tablet_bootstrap.cc:492] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:11:01.787133  4900 tablet_bootstrap.cc:492] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:11:01.787395  4900 ts_tablet_manager.cc:1367] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240501 08:11:01.788286  4900 raft_consensus.cc:348] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.788657  4900 raft_consensus.cc:374] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.788806  4900 raft_consensus.cc:729] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.789108  4900 consensus_queue.cc:260] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.790179  4901 tablet_bootstrap.cc:654] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.788972  4899 tablet_bootstrap.cc:654] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.790558  4900 ts_tablet_manager.cc:1398] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:11:01.792426  4896 tablet_bootstrap.cc:492] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:11:01.792630  4896 ts_tablet_manager.cc:1367] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.000s
I20240501 08:11:01.793496  4899 tablet_bootstrap.cc:492] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:11:01.793871  4899 ts_tablet_manager.cc:1367] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.000s
I20240501 08:11:01.794802  4899 raft_consensus.cc:348] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.795413  4899 raft_consensus.cc:374] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.795573  4899 raft_consensus.cc:729] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.795918  4899 consensus_queue.cc:260] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.793561  4896 raft_consensus.cc:348] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:01.796250  4896 raft_consensus.cc:374] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.796409  4896 raft_consensus.cc:729] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.796425  4897 tablet_bootstrap.cc:492] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:11:01.796591  4897 ts_tablet_manager.cc:1367] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.021s	user 0.006s	sys 0.000s
I20240501 08:11:01.796700  4896 consensus_queue.cc:260] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:01.796818  4899 ts_tablet_manager.cc:1398] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:11:01.797314  4896 ts_tablet_manager.cc:1398] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:11:01.797324  4897 raft_consensus.cc:348] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.797518  4897 raft_consensus.cc:374] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.797614  4897 raft_consensus.cc:729] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.797824  4897 consensus_queue.cc:260] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.799654  4897 ts_tablet_manager.cc:1398] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:11:01.800666  4898 tablet_bootstrap.cc:492] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:11:01.800875  4898 ts_tablet_manager.cc:1367] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.003s
I20240501 08:11:01.801705  4898 raft_consensus.cc:348] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.801959  4898 raft_consensus.cc:374] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.802100  4898 raft_consensus.cc:729] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.802268  4901 tablet_bootstrap.cc:492] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:11:01.802359  4898 consensus_queue.cc:260] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.802479  4901 ts_tablet_manager.cc:1367] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.000s
I20240501 08:11:01.803061  4898 ts_tablet_manager.cc:1398] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:11:01.803397  4898 tablet_bootstrap.cc:492] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:11:01.803413  4901 raft_consensus.cc:348] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.803683  4901 raft_consensus.cc:374] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.803826  4901 raft_consensus.cc:729] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.804358  4901 consensus_queue.cc:260] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.805073  4898 tablet_bootstrap.cc:654] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.806269  4901 ts_tablet_manager.cc:1398] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:11:01.806645  4901 tablet_bootstrap.cc:492] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:11:01.809104  4901 tablet_bootstrap.cc:654] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.812278  4896 tablet_bootstrap.cc:492] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:11:01.814750  4896 tablet_bootstrap.cc:654] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.816442  4898 tablet_bootstrap.cc:492] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:11:01.816892  4898 ts_tablet_manager.cc:1367] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.009s
I20240501 08:11:01.817091  4901 tablet_bootstrap.cc:492] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:11:01.817322  4901 ts_tablet_manager.cc:1367] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.007s
I20240501 08:11:01.817744  4898 raft_consensus.cc:348] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:01.818069  4898 raft_consensus.cc:374] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.818194  4898 raft_consensus.cc:729] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.818534  4898 consensus_queue.cc:260] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:01.819073  4901 raft_consensus.cc:348] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:01.819716  4901 raft_consensus.cc:374] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.819834  4901 raft_consensus.cc:729] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.819936  4896 tablet_bootstrap.cc:492] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:11:01.820160  4896 ts_tablet_manager.cc:1367] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240501 08:11:01.820171  4901 consensus_queue.cc:260] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:01.820852  4901 ts_tablet_manager.cc:1398] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:11:01.820930  4896 raft_consensus.cc:348] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.821158  4896 raft_consensus.cc:374] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.821242  4896 raft_consensus.cc:729] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.821477  4896 consensus_queue.cc:260] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.821879  4898 ts_tablet_manager.cc:1398] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:11:01.822108  4896 ts_tablet_manager.cc:1398] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:11:01.825790  4792 raft_consensus.cc:482] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:01.825922  4792 raft_consensus.cc:504] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.826613  4792 leader_election.cc:288] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:11:01.826911  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988163c67d974e01b512a163da9790c3" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:11:01.827106  3751 raft_consensus.cc:2454] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:11:01.827072  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988163c67d974e01b512a163da9790c3" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:11:01.827286  3226 raft_consensus.cc:2454] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 0.
I20240501 08:11:01.827458  3425 leader_election.cc:302] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:11:01.827715  4890 raft_consensus.cc:2781] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:01.827800  4890 raft_consensus.cc:482] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:01.827906  4890 raft_consensus.cc:3037] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.829854  4890 raft_consensus.cc:504] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.830477  4890 leader_election.cc:288] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Requested vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767)
I20240501 08:11:01.830842  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988163c67d974e01b512a163da9790c3" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:11:01.831056  3226 raft_consensus.cc:3037] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.831041  3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988163c67d974e01b512a163da9790c3" candidate_uuid: "97336c9b436644a798689a443b2f272e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:11:01.831257  3751 raft_consensus.cc:3037] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.833884  3226 raft_consensus.cc:2454] T 988163c67d974e01b512a163da9790c3 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:11:01.834049  3751 raft_consensus.cc:2454] T 988163c67d974e01b512a163da9790c3 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97336c9b436644a798689a443b2f272e in term 1.
I20240501 08:11:01.834491  3425 leader_election.cc:302] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:11:01.834735  4890 raft_consensus.cc:2781] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:01.834879  4890 raft_consensus.cc:686] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [term 1 LEADER]: Becoming Leader. State: Replica: 97336c9b436644a798689a443b2f272e, State: Running, Role: LEADER
I20240501 08:11:01.835127  4890 consensus_queue.cc:237] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } }
I20240501 08:11:01.837879  3079 catalog_manager.cc:5484] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e reported cstate change: term changed from 0 to 1, leader changed from <none> to 97336c9b436644a798689a443b2f272e (127.2.248.131). New cstate: current_term: 1 leader_uuid: "97336c9b436644a798689a443b2f272e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:01.973119  4824 raft_consensus.cc:482] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:01.973316  4824 raft_consensus.cc:504] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.973968  4824 leader_election.cc:288] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:11:01.974537  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58f1eebd5da4a9b8d2ce592f24bc5d2" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:11:01.974530  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58f1eebd5da4a9b8d2ce592f24bc5d2" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:11:01.974756  3620 raft_consensus.cc:2454] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:11:01.974778  3358 raft_consensus.cc:2454] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:11:01.975185  3817 leader_election.cc:302] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:11:01.975450  4824 raft_consensus.cc:2781] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:01.975705  4824 raft_consensus.cc:482] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:01.975807  4824 raft_consensus.cc:3037] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.977840  4824 raft_consensus.cc:504] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.978358  4824 leader_election.cc:288] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267), 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957)
I20240501 08:11:01.978806  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58f1eebd5da4a9b8d2ce592f24bc5d2" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:11:01.978816  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58f1eebd5da4a9b8d2ce592f24bc5d2" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:11:01.978963  3620 raft_consensus.cc:3037] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.978974  3358 raft_consensus.cc:3037] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.980816  3358 raft_consensus.cc:2454] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:11:01.980824  3620 raft_consensus.cc:2454] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:11:01.981178  3817 leader_election.cc:302] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:11:01.981418  4824 raft_consensus.cc:2781] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:01.981801  4824 raft_consensus.cc:686] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:11:01.982096  4824 consensus_queue.cc:237] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } }
I20240501 08:11:01.984813  3079 catalog_manager.cc:5484] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:02.087571  4915 raft_consensus.cc:482] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:02.087733  4915 raft_consensus.cc:504] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:02.088626  4915 leader_election.cc:288] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:11:02.089152  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cf58d92eda427d8c0e88d17a8e0c92" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" is_pre_election: true
I20240501 08:11:02.089299  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cf58d92eda427d8c0e88d17a8e0c92" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:11:02.089433  3226 raft_consensus.cc:2454] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:11:02.089565  3489 raft_consensus.cc:2454] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 0.
I20240501 08:11:02.089843  3687 leader_election.cc:302] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:11:02.090139  4915 raft_consensus.cc:2781] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:02.090258  4915 raft_consensus.cc:482] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:02.090330  4915 raft_consensus.cc:3037] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.092075  4915 raft_consensus.cc:504] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:02.092577  4915 leader_election.cc:288] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Requested vote from peers c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:11:02.092964  3489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cf58d92eda427d8c0e88d17a8e0c92" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:11:02.092972  3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89cf58d92eda427d8c0e88d17a8e0c92" candidate_uuid: "b526e10277944ab0b4744f9de18a8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5"
I20240501 08:11:02.093133  3489 raft_consensus.cc:3037] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.093144  3226 raft_consensus.cc:3037] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.094810  3489 raft_consensus.cc:2454] T 89cf58d92eda427d8c0e88d17a8e0c92 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:11:02.094869  3226 raft_consensus.cc:2454] T 89cf58d92eda427d8c0e88d17a8e0c92 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b526e10277944ab0b4744f9de18a8771 in term 1.
I20240501 08:11:02.095166  3686 leader_election.cc:302] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97336c9b436644a798689a443b2f272e, b526e10277944ab0b4744f9de18a8771; no voters: 
I20240501 08:11:02.095497  4915 raft_consensus.cc:2781] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:02.095636  4915 raft_consensus.cc:686] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [term 1 LEADER]: Becoming Leader. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Running, Role: LEADER
I20240501 08:11:02.095873  4915 consensus_queue.cc:237] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } }
I20240501 08:11:02.098608  3079 catalog_manager.cc:5484] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 reported cstate change: term changed from 0 to 1, leader changed from <none> to b526e10277944ab0b4744f9de18a8771 (127.2.248.133). New cstate: current_term: 1 leader_uuid: "b526e10277944ab0b4744f9de18a8771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:02.272883  4890 consensus_queue.cc:1035] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.279093  4792 consensus_queue.cc:1035] T 988163c67d974e01b512a163da9790c3 P 97336c9b436644a798689a443b2f272e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.370285  4913 consensus_queue.cc:1035] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.376452  4913 consensus_queue.cc:1035] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:02.493548  4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:02.493858  4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:02.501446  4917 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 08:11:02.576516  4915 consensus_queue.cc:1035] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:02.583379  4915 consensus_queue.cc:1035] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 269.099 ms
  time per row: 0.13455 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:11:03.875432  3078 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35448:
name: "rebalance_test_table_11"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:11:03.884577  3338 tablet_service.cc:1467] Processing CreateTablet for tablet 82d2584acd244a7891df0f197c0e28fb (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.885022  3338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d2584acd244a7891df0f197c0e28fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.884991  3600 tablet_service.cc:1467] Processing CreateTablet for tablet 82d2584acd244a7891df0f197c0e28fb (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.885421  3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d2584acd244a7891df0f197c0e28fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.885901  3730 tablet_service.cc:1467] Processing CreateTablet for tablet 47b593452d4e47b6856bd3046f98f033 (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.886269  3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b593452d4e47b6856bd3046f98f033. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.886422  3206 tablet_service.cc:1467] Processing CreateTablet for tablet 47b593452d4e47b6856bd3046f98f033 (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.886801  3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b593452d4e47b6856bd3046f98f033. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.888998  3468 tablet_service.cc:1467] Processing CreateTablet for tablet 47b593452d4e47b6856bd3046f98f033 (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.889374  3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b593452d4e47b6856bd3046f98f033. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.891876  3337 tablet_service.cc:1467] Processing CreateTablet for tablet 313e015a1fcc49458d18ff3b88581041 (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.892254  3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313e015a1fcc49458d18ff3b88581041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.895301  3861 tablet_service.cc:1467] Processing CreateTablet for tablet 82d2584acd244a7891df0f197c0e28fb (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.895696  3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d2584acd244a7891df0f197c0e28fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.896471  3599 tablet_service.cc:1467] Processing CreateTablet for tablet 313e015a1fcc49458d18ff3b88581041 (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.896466  3862 tablet_service.cc:1467] Processing CreateTablet for tablet 313e015a1fcc49458d18ff3b88581041 (DEFAULT_TABLE table=rebalance_test_table_11 [id=799444e5907d4f6aae8b71a430dcca29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.896696  4991 tablet_bootstrap.cc:492] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771: Bootstrap starting.
I20240501 08:11:03.896793  3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313e015a1fcc49458d18ff3b88581041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.896793  3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313e015a1fcc49458d18ff3b88581041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.897112  4989 tablet_bootstrap.cc:492] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:11:03.898187  4991 tablet_bootstrap.cc:654] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.898605  4989 tablet_bootstrap.cc:654] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.911427  4989 tablet_bootstrap.cc:492] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:11:03.911765  4989 ts_tablet_manager.cc:1367] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.015s	user 0.000s	sys 0.012s
I20240501 08:11:03.912868  4989 raft_consensus.cc:348] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.913321  4989 raft_consensus.cc:374] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.913503  4989 raft_consensus.cc:729] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.913878  4989 consensus_queue.cc:260] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.914664  4989 ts_tablet_manager.cc:1398] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:11:03.917830  4996 tablet_bootstrap.cc:492] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:11:03.920581  4996 tablet_bootstrap.cc:654] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.921442  4989 tablet_bootstrap.cc:492] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878: Bootstrap starting.
I20240501 08:11:03.923478  4990 tablet_bootstrap.cc:492] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:11:03.923651  4989 tablet_bootstrap.cc:654] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.925310  4990 tablet_bootstrap.cc:654] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.925777  4994 tablet_bootstrap.cc:492] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e: Bootstrap starting.
I20240501 08:11:03.926585  4995 tablet_bootstrap.cc:492] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Bootstrap starting.
I20240501 08:11:03.928289  4991 tablet_bootstrap.cc:492] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771: No bootstrap required, opened a new log
I20240501 08:11:03.928534  4996 tablet_bootstrap.cc:492] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:11:03.928539  4991 ts_tablet_manager.cc:1367] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.014s
I20240501 08:11:03.928620  4995 tablet_bootstrap.cc:654] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.928685  4996 ts_tablet_manager.cc:1367] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240501 08:11:03.929536  4996 raft_consensus.cc:348] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.929746  4996 raft_consensus.cc:374] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.929837  4996 raft_consensus.cc:729] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.929944  4991 raft_consensus.cc:348] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.930053  4996 consensus_queue.cc:260] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.930400  4991 raft_consensus.cc:374] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.930653  4991 raft_consensus.cc:729] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b526e10277944ab0b4744f9de18a8771, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.930707  4996 ts_tablet_manager.cc:1398] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:03.931035  4996 tablet_bootstrap.cc:492] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0: Bootstrap starting.
I20240501 08:11:03.931038  4991 consensus_queue.cc:260] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.932709  4994 tablet_bootstrap.cc:654] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.934010  4996 tablet_bootstrap.cc:654] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.934517  4991 ts_tablet_manager.cc:1398] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 08:11:03.936327  4989 tablet_bootstrap.cc:492] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878: No bootstrap required, opened a new log
I20240501 08:11:03.936538  4989 ts_tablet_manager.cc:1367] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.002s
I20240501 08:11:03.937287  4989 raft_consensus.cc:348] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.937565  4989 raft_consensus.cc:374] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.937700  4989 raft_consensus.cc:729] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1979c6dfa70f42699a57e8210a28b878, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.938300  4990 tablet_bootstrap.cc:492] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:11:03.937964  4989 consensus_queue.cc:260] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.939141  4995 tablet_bootstrap.cc:492] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: No bootstrap required, opened a new log
I20240501 08:11:03.939358  4995 ts_tablet_manager.cc:1367] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.000s
I20240501 08:11:03.940352  4989 ts_tablet_manager.cc:1398] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:11:03.940600  4990 ts_tablet_manager.cc:1367] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.000s
I20240501 08:11:03.940359  4995 raft_consensus.cc:348] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.941674  4990 raft_consensus.cc:348] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.941839  4990 raft_consensus.cc:374] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.941959  4995 raft_consensus.cc:374] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.942061  4995 raft_consensus.cc:729] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.942323  4996 tablet_bootstrap.cc:492] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0: No bootstrap required, opened a new log
I20240501 08:11:03.942276  4995 consensus_queue.cc:260] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.942453  4996 ts_tablet_manager.cc:1367] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.005s
I20240501 08:11:03.943101  4996 raft_consensus.cc:348] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.943313  4996 raft_consensus.cc:374] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.943380  4996 raft_consensus.cc:729] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.943595  4996 consensus_queue.cc:260] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.944070  4996 ts_tablet_manager.cc:1398] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:11:03.944633  4995 ts_tablet_manager.cc:1398] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:11:03.945729  4994 tablet_bootstrap.cc:492] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e: No bootstrap required, opened a new log
I20240501 08:11:03.945874  4994 ts_tablet_manager.cc:1367] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.002s
I20240501 08:11:03.946677  4994 raft_consensus.cc:348] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.946899  4994 raft_consensus.cc:374] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.947012  4994 raft_consensus.cc:729] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97336c9b436644a798689a443b2f272e, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.947191  4994 consensus_queue.cc:260] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.947660  4990 raft_consensus.cc:729] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.947911  4990 consensus_queue.cc:260] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.948311  4994 ts_tablet_manager.cc:1398] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240501 08:11:03.949781  4990 ts_tablet_manager.cc:1398] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.002s
I20240501 08:11:03.950096  4990 tablet_bootstrap.cc:492] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14: Bootstrap starting.
I20240501 08:11:03.951740  4990 tablet_bootstrap.cc:654] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.953812  4990 tablet_bootstrap.cc:492] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14: No bootstrap required, opened a new log
I20240501 08:11:03.954041  4990 ts_tablet_manager.cc:1367] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 08:11:03.954839  4990 raft_consensus.cc:348] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.955099  4990 raft_consensus.cc:374] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.955193  4990 raft_consensus.cc:729] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.955459  4990 consensus_queue.cc:260] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:03.955895  4990 ts_tablet_manager.cc:1398] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:11:03.978111  4816 raft_consensus.cc:482] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:03.978264  4816 raft_consensus.cc:504] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.979116  4816 leader_election.cc:288] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:11:03.979442  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b593452d4e47b6856bd3046f98f033" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771" is_pre_election: true
I20240501 08:11:03.979609  3488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b593452d4e47b6856bd3046f98f033" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e" is_pre_election: true
I20240501 08:11:03.979691  3750 raft_consensus.cc:2454] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:11:03.979835  3488 raft_consensus.cc:2454] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 0.
I20240501 08:11:03.980096  3162 leader_election.cc:302] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:11:03.980571  4816 raft_consensus.cc:2781] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:03.980732  4816 raft_consensus.cc:482] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:03.980831  4816 raft_consensus.cc:3037] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:03.982975  4816 raft_consensus.cc:504] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.983471  4816 leader_election.cc:288] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Requested vote from peers b526e10277944ab0b4744f9de18a8771 (127.2.248.133:37767), 97336c9b436644a798689a443b2f272e (127.2.248.131:44211)
I20240501 08:11:03.983826  3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b593452d4e47b6856bd3046f98f033" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b526e10277944ab0b4744f9de18a8771"
I20240501 08:11:03.984051  3750 raft_consensus.cc:3037] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:03.984040  3488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b593452d4e47b6856bd3046f98f033" candidate_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97336c9b436644a798689a443b2f272e"
I20240501 08:11:03.984268  3488 raft_consensus.cc:3037] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:03.986832  3750 raft_consensus.cc:2454] T 47b593452d4e47b6856bd3046f98f033 P b526e10277944ab0b4744f9de18a8771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:11:03.986848  3488 raft_consensus.cc:2454] T 47b593452d4e47b6856bd3046f98f033 P 97336c9b436644a798689a443b2f272e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cc7544bb6d4ee99fb2afb0bdbd71e5 in term 1.
I20240501 08:11:03.987208  3162 leader_election.cc:302] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b526e10277944ab0b4744f9de18a8771, c5cc7544bb6d4ee99fb2afb0bdbd71e5; no voters: 
I20240501 08:11:03.987514  4816 raft_consensus.cc:2781] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:03.987823  4816 raft_consensus.cc:686] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [term 1 LEADER]: Becoming Leader. State: Replica: c5cc7544bb6d4ee99fb2afb0bdbd71e5, State: Running, Role: LEADER
I20240501 08:11:03.988147  4816 consensus_queue.cc:237] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } }
I20240501 08:11:03.991001  3079 catalog_manager.cc:5484] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cc7544bb6d4ee99fb2afb0bdbd71e5" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 32925 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:04.002779  4980 raft_consensus.cc:482] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:04.002933  4980 raft_consensus.cc:504] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:04.004145  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2584acd244a7891df0f197c0e28fb" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:11:04.004088  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2584acd244a7891df0f197c0e28fb" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0" is_pre_election: true
I20240501 08:11:04.004395  3358 raft_consensus.cc:2454] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:11:04.004501  3882 raft_consensus.cc:2454] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 0.
I20240501 08:11:04.003595  4980 leader_election.cc:288] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:11:04.004921  3556 leader_election.cc:302] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 29eea43c04ae49edb64ba8803bf09b14; no voters: 
I20240501 08:11:04.005293  4980 raft_consensus.cc:2781] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:04.005409  4980 raft_consensus.cc:482] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:04.005479  4980 raft_consensus.cc:3037] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:04.007150  4980 raft_consensus.cc:504] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:04.007650  4980 leader_election.cc:288] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134:37605)
I20240501 08:11:04.008024  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2584acd244a7891df0f197c0e28fb" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:11:04.008237  3358 raft_consensus.cc:3037] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:04.008167  3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2584acd244a7891df0f197c0e28fb" candidate_uuid: "29eea43c04ae49edb64ba8803bf09b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6deb22ef32e2476cb56efabd79a948c0"
I20240501 08:11:04.008375  3882 raft_consensus.cc:3037] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:04.010135  3358 raft_consensus.cc:2454] T 82d2584acd244a7891df0f197c0e28fb P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:11:04.010183  3882 raft_consensus.cc:2454] T 82d2584acd244a7891df0f197c0e28fb P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29eea43c04ae49edb64ba8803bf09b14 in term 1.
I20240501 08:11:04.010583  3553 leader_election.cc:302] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:11:04.010857  4980 raft_consensus.cc:2781] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:04.011411  4980 raft_consensus.cc:686] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [term 1 LEADER]: Becoming Leader. State: Replica: 29eea43c04ae49edb64ba8803bf09b14, State: Running, Role: LEADER
I20240501 08:11:04.011780  4980 consensus_queue.cc:237] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:04.014760  3079 catalog_manager.cc:5484] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132). New cstate: current_term: 1 leader_uuid: "29eea43c04ae49edb64ba8803bf09b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:04.118216  4824 raft_consensus.cc:482] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:04.118427  4824 raft_consensus.cc:504] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:04.119143  4824 leader_election.cc:288] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:11:04.119593  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313e015a1fcc49458d18ff3b88581041" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878" is_pre_election: true
I20240501 08:11:04.119689  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313e015a1fcc49458d18ff3b88581041" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14" is_pre_election: true
I20240501 08:11:04.119855  3358 raft_consensus.cc:2454] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:11:04.119907  3620 raft_consensus.cc:2454] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 0.
I20240501 08:11:04.120282  3818 leader_election.cc:302] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1979c6dfa70f42699a57e8210a28b878, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:11:04.120672  4824 raft_consensus.cc:2781] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:04.120807  4824 raft_consensus.cc:482] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:04.120926  4824 raft_consensus.cc:3037] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:04.122673  4824 raft_consensus.cc:504] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:04.123173  4824 leader_election.cc:288] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957), 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267)
I20240501 08:11:04.123513  3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313e015a1fcc49458d18ff3b88581041" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29eea43c04ae49edb64ba8803bf09b14"
I20240501 08:11:04.123716  3620 raft_consensus.cc:3037] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:04.123697  3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313e015a1fcc49458d18ff3b88581041" candidate_uuid: "6deb22ef32e2476cb56efabd79a948c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1979c6dfa70f42699a57e8210a28b878"
I20240501 08:11:04.123900  3358 raft_consensus.cc:3037] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:04.126379  3620 raft_consensus.cc:2454] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:11:04.126600  3358 raft_consensus.cc:2454] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6deb22ef32e2476cb56efabd79a948c0 in term 1.
I20240501 08:11:04.126782  3817 leader_election.cc:302] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29eea43c04ae49edb64ba8803bf09b14, 6deb22ef32e2476cb56efabd79a948c0; no voters: 
I20240501 08:11:04.127094  4824 raft_consensus.cc:2781] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:04.127254  4824 raft_consensus.cc:686] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [term 1 LEADER]: Becoming Leader. State: Replica: 6deb22ef32e2476cb56efabd79a948c0, State: Running, Role: LEADER
I20240501 08:11:04.127496  4824 consensus_queue.cc:237] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } }
I20240501 08:11:04.130913  3079 catalog_manager.cc:5484] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6deb22ef32e2476cb56efabd79a948c0 (127.2.248.134). New cstate: current_term: 1 leader_uuid: "6deb22ef32e2476cb56efabd79a948c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:04.459967  4816 consensus_queue.cc:1035] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97336c9b436644a798689a443b2f272e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:04.460495  5009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:04.460844  5009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:04.464269  4816 consensus_queue.cc:1035] T 47b593452d4e47b6856bd3046f98f033 P c5cc7544bb6d4ee99fb2afb0bdbd71e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b526e10277944ab0b4744f9de18a8771" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:04.470253  5009 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:04.498246  4980 consensus_queue.cc:1035] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:04.504818  5006 consensus_queue.cc:1035] T 82d2584acd244a7891df0f197c0e28fb P 29eea43c04ae49edb64ba8803bf09b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6deb22ef32e2476cb56efabd79a948c0" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
I20240501 08:11:04.607623  3620 raft_consensus.cc:1261] T 313e015a1fcc49458d18ff3b88581041 P 29eea43c04ae49edb64ba8803bf09b14 [term 1 FOLLOWER]: Refusing update from remote peer 6deb22ef32e2476cb56efabd79a948c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:04.608389  4824 consensus_queue.cc:1035] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29eea43c04ae49edb64ba8803bf09b14" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:04.622592  3358 raft_consensus.cc:1261] T 313e015a1fcc49458d18ff3b88581041 P 1979c6dfa70f42699a57e8210a28b878 [term 1 FOLLOWER]: Refusing update from remote peer 6deb22ef32e2476cb56efabd79a948c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:04.623368  4824 consensus_queue.cc:1035] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1979c6dfa70f42699a57e8210a28b878" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:04.668802  5028 mvcc.cc:205] Tried to move back new op lower bound from 7022801160626913280 to 7022801158669631488. Current Snapshot: MvccSnapshot[applied={T|T < 7022801160626913280}]
I20240501 08:11:04.715929  5034 mvcc.cc:205] Tried to move back new op lower bound from 7022801160626913280 to 7022801158669631488. Current Snapshot: MvccSnapshot[applied={T|T < 7022801160626913280}]

INSERT report
    rows total: 2000
    time total: 301.629 ms
  time per row: 0.150815 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:11:06.349237  3337 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:06.372047  3862 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:11:06.377570  3730 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:06.400522  3206 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:06.401111  3600 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:06.434178  3468 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:06.632419  3337 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:06.639848  3600 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:06.650157  3862 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:11:06.659166  3730 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:06.672240  3468 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:06.679862  3206 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:06.944849  3337 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:06.985235  3730 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:06.992744  3206 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:07.004434  3600 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:07.035780  3468 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:07.039420  3862 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:11:07.238216  3337 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:07.238441  3862 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:11:07.240841  3600 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:07.250988  3468 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:07.269979  3730 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:07.298527  3206 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:07.510849  3337 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:07.511000  3862 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:11:07.515869  3600 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:07.538232  3468 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:07.542650  3730 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:07.544463  3206 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:07.935870  3337 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:07.940753  3600 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:07.949505  3862 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:11:07.965404  3206 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:07.966739  3468 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:07.968271  3730 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 3079d9ef376b44d28c2fee88d70d0d5a | 127.2.248.190:42765 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                    Value                                                                                     |         Tags          |         Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                         | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                            | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                         | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                         | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                         | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                           | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                          | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:43817 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1979c6dfa70f42699a57e8210a28b878 | 127.2.248.130:33957 | HEALTHY | /A       |       2        |       0
 29eea43c04ae49edb64ba8803bf09b14 | 127.2.248.132:37267 | HEALTHY | /B       |       6        |       0
 6deb22ef32e2476cb56efabd79a948c0 | 127.2.248.134:37605 | HEALTHY | /C       |      11        |       0
 97336c9b436644a798689a443b2f272e | 127.2.248.131:44211 | HEALTHY | /B       |       7        |       0
 b526e10277944ab0b4744f9de18a8771 | 127.2.248.133:37767 | HEALTHY | /C       |       5        |       0
 c5cc7544bb6d4ee99fb2afb0bdbd71e5 | 127.2.248.129:32925 | HEALTHY | /A       |       5        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 /C       |       2
 /A       |       2
 /B       |       2

Unusual flags for Tablet Server:
                   Flag                   |                                                                                  Value                                                                                   |         Tags          |      Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                      | experimental          | all 6 server(s) checked
 local_ip_for_outbound_sockets            | 127.2.248.129                                                                                                                                                            | experimental          | 127.2.248.129:32925
 local_ip_for_outbound_sockets            | 127.2.248.130                                                                                                                                                            | experimental          | 127.2.248.130:33957
 local_ip_for_outbound_sockets            | 127.2.248.131                                                                                                                                                            | experimental          | 127.2.248.131:44211
 local_ip_for_outbound_sockets            | 127.2.248.132                                                                                                                                                            | experimental          | 127.2.248.132:37267
 local_ip_for_outbound_sockets            | 127.2.248.133                                                                                                                                                            | experimental          | 127.2.248.133:37767
 local_ip_for_outbound_sockets            | 127.2.248.134                                                                                                                                                            | experimental          | 127.2.248.134:37605
 never_fsync                              | true                                                                                                                                                                     | unsafe,advanced       | all 6 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                        | unsafe,hidden         | all 6 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                     | experimental,advanced | all 6 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                     | unsafe                | all 6 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                       | hidden                | all 6 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.2.248.129:32925
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.2.248.130:33957
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.2.248.131:44211
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.2.248.132:37267
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.2.248.133:37767
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.2.248.134:37605

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:43817 | all 6 server(s) checked
 time_source         | builtin             | all 6 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 7 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name           | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
 rebalance_test_table_0  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_10 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_11 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_5  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_6  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_7  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_8  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_9  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 18
 First Quartile | 18
 Median         | 18
 Third Quartile | 18
 Maximum        | 18

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 6
 Tables         | 12
 Tablets        | 36
 Replicas       | 108

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 08:11:08.052011  3042 log_verifier.cc:126] Checking tablet 01562e1616fb4a44af73d56f6b75b8e6
I20240501 08:11:08.089222  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 01562e1616fb4a44af73d56f6b75b8e6
I20240501 08:11:08.089376  3042 log_verifier.cc:126] Checking tablet 1002d7b484654f908eb6732dabb63e4f
I20240501 08:11:08.124465  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1002d7b484654f908eb6732dabb63e4f
I20240501 08:11:08.124562  3042 log_verifier.cc:126] Checking tablet 1a434e9552e249f9b09853770852058f
I20240501 08:11:08.160233  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1a434e9552e249f9b09853770852058f
I20240501 08:11:08.160353  3042 log_verifier.cc:126] Checking tablet 1ae945fb63414d88b8ec336244aeb55e
I20240501 08:11:08.200436  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1ae945fb63414d88b8ec336244aeb55e
I20240501 08:11:08.200556  3042 log_verifier.cc:126] Checking tablet 313e015a1fcc49458d18ff3b88581041
I20240501 08:11:08.240010  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 313e015a1fcc49458d18ff3b88581041
I20240501 08:11:08.240140  3042 log_verifier.cc:126] Checking tablet 46e45c482291420da0989bb628d3485d
I20240501 08:11:08.278291  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 46e45c482291420da0989bb628d3485d
I20240501 08:11:08.278398  3042 log_verifier.cc:126] Checking tablet 47b593452d4e47b6856bd3046f98f033
I20240501 08:11:08.316802  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 47b593452d4e47b6856bd3046f98f033
I20240501 08:11:08.316905  3042 log_verifier.cc:126] Checking tablet 500ec289b2d84a6989c1745fab74c5d4
I20240501 08:11:08.355559  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 500ec289b2d84a6989c1745fab74c5d4
I20240501 08:11:08.355710  3042 log_verifier.cc:126] Checking tablet 5bc4623bbc0e4a20944a3f863e311128
I20240501 08:11:08.395978  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5bc4623bbc0e4a20944a3f863e311128
I20240501 08:11:08.396147  3042 log_verifier.cc:126] Checking tablet 5c5d8eded44d4c85b097e7d3cfa153eb
I20240501 08:11:08.446492  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5c5d8eded44d4c85b097e7d3cfa153eb
I20240501 08:11:08.446614  3042 log_verifier.cc:126] Checking tablet 6bf57c4a7fb443ab9527a779f91360cc
I20240501 08:11:08.485525  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6bf57c4a7fb443ab9527a779f91360cc
I20240501 08:11:08.485652  3042 log_verifier.cc:126] Checking tablet 6c7c327c7b794abca7cda496441e2ae6
I20240501 08:11:08.522740  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6c7c327c7b794abca7cda496441e2ae6
I20240501 08:11:08.522842  3042 log_verifier.cc:126] Checking tablet 709f953859ea4186b38c911c30b5f9b0
I20240501 08:11:08.559407  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 709f953859ea4186b38c911c30b5f9b0
I20240501 08:11:08.559523  3042 log_verifier.cc:126] Checking tablet 75ed8727af504012bf9d5242deec4fd6
I20240501 08:11:08.599637  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 75ed8727af504012bf9d5242deec4fd6
I20240501 08:11:08.599750  3042 log_verifier.cc:126] Checking tablet 82d2584acd244a7891df0f197c0e28fb
I20240501 08:11:08.647562  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 82d2584acd244a7891df0f197c0e28fb
I20240501 08:11:08.647711  3042 log_verifier.cc:126] Checking tablet 867e0435e12a4255a9aaa7d42dbe45a7
I20240501 08:11:08.685753  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 867e0435e12a4255a9aaa7d42dbe45a7
I20240501 08:11:08.685868  3042 log_verifier.cc:126] Checking tablet 89cf58d92eda427d8c0e88d17a8e0c92
I20240501 08:11:08.723471  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 89cf58d92eda427d8c0e88d17a8e0c92
I20240501 08:11:08.723608  3042 log_verifier.cc:126] Checking tablet 89d4132a23724130b25d6bdf7438c315
I20240501 08:11:08.770644  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 89d4132a23724130b25d6bdf7438c315
I20240501 08:11:08.770761  3042 log_verifier.cc:126] Checking tablet 8fc5921f69664791a6d20c48440e6bf9
I20240501 08:11:08.808923  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8fc5921f69664791a6d20c48440e6bf9
I20240501 08:11:08.809037  3042 log_verifier.cc:126] Checking tablet 988163c67d974e01b512a163da9790c3
I20240501 08:11:08.849336  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 988163c67d974e01b512a163da9790c3
I20240501 08:11:08.849462  3042 log_verifier.cc:126] Checking tablet a42dd63dc2c747198b0e78bb3fa1969b
I20240501 08:11:08.887161  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a42dd63dc2c747198b0e78bb3fa1969b
I20240501 08:11:08.887295  3042 log_verifier.cc:126] Checking tablet ad0a4d86e4a145b3a45f43e6b2ffc7da
I20240501 08:11:08.924492  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ad0a4d86e4a145b3a45f43e6b2ffc7da
I20240501 08:11:08.924607  3042 log_verifier.cc:126] Checking tablet aef6c6e24d184079a8c092f20e010f85
I20240501 08:11:08.963778  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet aef6c6e24d184079a8c092f20e010f85
I20240501 08:11:08.963889  3042 log_verifier.cc:126] Checking tablet b29d7b5336644659a0c3f7beef22d5b9
I20240501 08:11:09.001317  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b29d7b5336644659a0c3f7beef22d5b9
I20240501 08:11:09.001569  3042 log_verifier.cc:126] Checking tablet b3a51a508c09466ebc399b8441cd3f64
I20240501 08:11:09.041564  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b3a51a508c09466ebc399b8441cd3f64
I20240501 08:11:09.041669  3042 log_verifier.cc:126] Checking tablet b6d31a9a1fe048dda14ce04e0050644e
I20240501 08:11:09.080401  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b6d31a9a1fe048dda14ce04e0050644e
I20240501 08:11:09.080505  3042 log_verifier.cc:126] Checking tablet b8ac230309f0427dab6698aba1fb1e55
I20240501 08:11:09.116508  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b8ac230309f0427dab6698aba1fb1e55
I20240501 08:11:09.116619  3042 log_verifier.cc:126] Checking tablet bbda0193ab414fdeaa1748c23e3c889c
I20240501 08:11:09.153853  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bbda0193ab414fdeaa1748c23e3c889c
I20240501 08:11:09.153981  3042 log_verifier.cc:126] Checking tablet c575114ae3b04353817dd802b07d5011
I20240501 08:11:09.193113  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c575114ae3b04353817dd802b07d5011
I20240501 08:11:09.193244  3042 log_verifier.cc:126] Checking tablet c7d7f4724716476990b2303cafa4ade4
I20240501 08:11:09.231979  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c7d7f4724716476990b2303cafa4ade4
I20240501 08:11:09.232098  3042 log_verifier.cc:126] Checking tablet d66bace0bf96429eab97117c872963e1
I20240501 08:11:09.271549  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d66bace0bf96429eab97117c872963e1
I20240501 08:11:09.271689  3042 log_verifier.cc:126] Checking tablet d6c7063056a34e33a2b21f1934910246
I20240501 08:11:09.319036  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d6c7063056a34e33a2b21f1934910246
I20240501 08:11:09.319170  3042 log_verifier.cc:126] Checking tablet e58f1eebd5da4a9b8d2ce592f24bc5d2
I20240501 08:11:09.359547  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e58f1eebd5da4a9b8d2ce592f24bc5d2
I20240501 08:11:09.359700  3042 log_verifier.cc:126] Checking tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a
I20240501 08:11:09.398885  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet eafcf5e6c2af4df9aaf4107dcd5c9b7a
I20240501 08:11:09.399001  3042 log_verifier.cc:126] Checking tablet edb2ba21e0df4769b9ca97b451c36d1b
I20240501 08:11:09.442171  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet edb2ba21e0df4769b9ca97b451c36d1b
I20240501 08:11:09.442297  3042 log_verifier.cc:126] Checking tablet f6212c5cba30496c80c61a4014be2b9b
I20240501 08:11:09.484159  3042 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f6212c5cba30496c80c61a4014be2b9b
I20240501 08:11:09.504603  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 3144
I20240501 08:11:09.517010  3265 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:09.655886  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 3144
W20240501 08:11:09.680913  3685 connection.cc:535] server connection from 127.2.248.129:48805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:09.681679  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 3275
W20240501 08:11:09.686658  3425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111)
I20240501 08:11:09.687522  3397 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:11:09.688454  3425 consensus_peers.cc:485] T d6c7063056a34e33a2b21f1934910246 P 97336c9b436644a798689a443b2f272e -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.736073  3425 consensus_peers.cc:485] T 8fc5921f69664791a6d20c48440e6bf9 P 97336c9b436644a798689a443b2f272e -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.738077  3425 consensus_peers.cc:485] T 1ae945fb63414d88b8ec336244aeb55e P 97336c9b436644a798689a443b2f272e -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.773334  3425 consensus_peers.cc:485] T 1a434e9552e249f9b09853770852058f P 97336c9b436644a798689a443b2f272e -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.801595  3425 consensus_peers.cc:485] T 89d4132a23724130b25d6bdf7438c315 P 97336c9b436644a798689a443b2f272e -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:09.821939  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 3275
W20240501 08:11:09.853399  3069 connection.cc:535] server connection from 127.2.248.130:43465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:09.853472  3818 connection.cc:535] client connection to 127.2.248.130:33957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:09.853734  3818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:09.853785  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 3406
W20240501 08:11:09.855422  3818 consensus_peers.cc:485] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.855548  3818 consensus_peers.cc:485] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.855634  3818 consensus_peers.cc:485] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.855729  3818 consensus_peers.cc:485] T 500ec289b2d84a6989c1745fab74c5d4 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.858625  3556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111)
W20240501 08:11:09.860044  3556 consensus_peers.cc:485] T b3a51a508c09466ebc399b8441cd3f64 P 29eea43c04ae49edb64ba8803bf09b14 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:09.860759  3528 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:11:09.864288  3687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111)
W20240501 08:11:09.866021  3687 consensus_peers.cc:485] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.875761  3687 consensus_peers.cc:485] T 709f953859ea4186b38c911c30b5f9b0 P b526e10277944ab0b4744f9de18a8771 -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.909806  3818 consensus_peers.cc:485] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.936486  3818 consensus_peers.cc:485] T e58f1eebd5da4a9b8d2ce592f24bc5d2 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.939280  3687 consensus_peers.cc:485] T 89cf58d92eda427d8c0e88d17a8e0c92 P b526e10277944ab0b4744f9de18a8771 -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:09.971544  3818 consensus_peers.cc:485] T 01562e1616fb4a44af73d56f6b75b8e6 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:09.989734  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 3406
W20240501 08:11:10.012277  3817 connection.cc:535] client connection to 127.2.248.131:44211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:10.012459  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 3537
W20240501 08:11:10.013545  3817 consensus_peers.cc:485] T d66bace0bf96429eab97117c872963e1 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 97336c9b436644a798689a443b2f272e (127.2.248.131:44211): Couldn't send request to peer 97336c9b436644a798689a443b2f272e. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.131:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:10.018127  3659 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:11:10.041918  3687 consensus_peers.cc:485] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.066038  3818 consensus_peers.cc:485] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.077219  3686 consensus_peers.cc:485] T 46e45c482291420da0989bb628d3485d P b526e10277944ab0b4744f9de18a8771 -> Peer 97336c9b436644a798689a443b2f272e (127.2.248.131:44211): Couldn't send request to peer 97336c9b436644a798689a443b2f272e. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.131:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.105218  3686 consensus_peers.cc:485] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 -> Peer 97336c9b436644a798689a443b2f272e (127.2.248.131:44211): Couldn't send request to peer 97336c9b436644a798689a443b2f272e. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.131:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.108888  3687 consensus_peers.cc:485] T b6d31a9a1fe048dda14ce04e0050644e P b526e10277944ab0b4744f9de18a8771 -> Peer c5cc7544bb6d4ee99fb2afb0bdbd71e5 (127.2.248.129:32925): Couldn't send request to peer c5cc7544bb6d4ee99fb2afb0bdbd71e5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:32925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.128619  3686 consensus_peers.cc:485] T 1002d7b484654f908eb6732dabb63e4f P b526e10277944ab0b4744f9de18a8771 -> Peer 97336c9b436644a798689a443b2f272e (127.2.248.131:44211): Couldn't send request to peer 97336c9b436644a798689a443b2f272e. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.131:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.131199  3818 consensus_peers.cc:485] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:10.152761  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 3537
I20240501 08:11:10.178261  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 3668
W20240501 08:11:10.179417  3818 consensus_peers.cc:485] T b29d7b5336644659a0c3f7beef22d5b9 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.180312  3817 consensus_peers.cc:485] T 867e0435e12a4255a9aaa7d42dbe45a7 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267): Couldn't send request to peer 29eea43c04ae49edb64ba8803bf09b14. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.132:37267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.180483  3817 consensus_peers.cc:485] T 75ed8727af504012bf9d5242deec4fd6 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267): Couldn't send request to peer 29eea43c04ae49edb64ba8803bf09b14. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.132:37267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:10.185357  3790 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:11:10.250212  3818 consensus_peers.cc:485] T c7d7f4724716476990b2303cafa4ade4 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 1979c6dfa70f42699a57e8210a28b878 (127.2.248.130:33957): Couldn't send request to peer 1979c6dfa70f42699a57e8210a28b878. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:33957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:10.288475  3817 consensus_peers.cc:485] T 6bf57c4a7fb443ab9527a779f91360cc P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267): Couldn't send request to peer 29eea43c04ae49edb64ba8803bf09b14. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.132:37267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:10.315956  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 3668
W20240501 08:11:10.319905  3817 consensus_peers.cc:485] T 313e015a1fcc49458d18ff3b88581041 P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267): Couldn't send request to peer 29eea43c04ae49edb64ba8803bf09b14. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.132:37267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:10.340301  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 3799
W20240501 08:11:10.344326  3817 consensus_peers.cc:485] T eafcf5e6c2af4df9aaf4107dcd5c9b7a P 6deb22ef32e2476cb56efabd79a948c0 -> Peer 29eea43c04ae49edb64ba8803bf09b14 (127.2.248.132:37267): Couldn't send request to peer 29eea43c04ae49edb64ba8803bf09b14. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.132:37267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:10.347103  3921 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:10.483701  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 3799
W20240501 08:11:10.503206  3071 connection.cc:535] server connection from 127.2.248.134:45443 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:10.503532  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 3054
I20240501 08:11:10.504421  3115 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:10.654043  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 3054
2024-05-01T08:11:10Z chronyd exiting
[       OK ] LocationAwareRebalancingBasicTest.Basic (32470 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (32470 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240501 08:11:10.804430  3042 test_util.cc:268] Using random seed: 126241196
I20240501 08:11:10.806131  3042 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:10.806202  3042 ts_itest-base.cc:113] --------------
I20240501 08:11:10.806236  3042 ts_itest-base.cc:114] 5 tablet servers
I20240501 08:11:10.806299  3042 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:11:10.806345  3042 ts_itest-base.cc:116] --------------
2024-05-01T08:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:10Z Disabled control of system clock
I20240501 08:11:10.839771  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:37453
--webserver_interface=127.2.248.190
--webserver_port=0
--builtin_ntp_servers=127.2.248.148:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.248.190:37453
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:11:11.152873  5405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:11.153143  5405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:11.153227  5405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:11.160125  5405 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:11.160252  5405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:11.160303  5405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:11.160357  5405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:11.160408  5405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:11.168743  5405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.248.190:37453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.248.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:11.169951  5405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:11.171664  5405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:11.182207  5410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.182839  5413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.183415  5411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:11.184149  5405 server_base.cc:1034] running on GCE node
I20240501 08:11:11.184873  5405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:11.186307  5405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:11.187508  5405 hybrid_clock.cc:648] HybridClock initialized: now 1714551071187456 us; error 51 us; skew 500 ppm
I20240501 08:11:11.187837  5405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:11.189978  5405 webserver.cc:460] Webserver started at http://127.2.248.190:32841/ using document root <none> and password file <none>
I20240501 08:11:11.190644  5405 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:11.190742  5405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:11.191061  5405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:11.193657  5405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3e087bedee884088baf210bc009f686b"
format_stamp: "Formatted at 2024-05-01 08:11:11 on dist-test-slave-0pdg"
I20240501 08:11:11.194403  5405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3e087bedee884088baf210bc009f686b"
format_stamp: "Formatted at 2024-05-01 08:11:11 on dist-test-slave-0pdg"
I20240501 08:11:11.199326  5405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20240501 08:11:11.202422  5419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:11.203680  5405 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:11:11.203832  5405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3e087bedee884088baf210bc009f686b"
format_stamp: "Formatted at 2024-05-01 08:11:11 on dist-test-slave-0pdg"
I20240501 08:11:11.204007  5405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:11.219226  5405 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:11.219972  5405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:11.220263  5405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:11.237947  5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.190:37453 every 8 connection(s)
I20240501 08:11:11.237922  5405 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.190:37453
I20240501 08:11:11.239382  5405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:11:11.242894  5471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:11.249306  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 5405
I20240501 08:11:11.249533  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:11:11.252516  5471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b: Bootstrap starting.
I20240501 08:11:11.255452  5471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:11.256790  5471 log.cc:824] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:11.259599  5471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b: No bootstrap required, opened a new log
I20240501 08:11:11.266492  5471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e087bedee884088baf210bc009f686b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 37453 } }
I20240501 08:11:11.266778  5471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:11.266850  5471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e087bedee884088baf210bc009f686b, State: Initialized, Role: FOLLOWER
I20240501 08:11:11.267357  5471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e087bedee884088baf210bc009f686b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 37453 } }
I20240501 08:11:11.267532  5471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:11.267616  5471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:11.267724  5471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:11.269690  5471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e087bedee884088baf210bc009f686b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 37453 } }
I20240501 08:11:11.270133  5471 leader_election.cc:302] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e087bedee884088baf210bc009f686b; no voters: 
I20240501 08:11:11.270668  5471 leader_election.cc:288] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:11:11.270802  5476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:11.271416  5476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [term 1 LEADER]: Becoming Leader. State: Replica: 3e087bedee884088baf210bc009f686b, State: Running, Role: LEADER
I20240501 08:11:11.271919  5476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e087bedee884088baf210bc009f686b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 37453 } }
I20240501 08:11:11.272485  5471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:11.274806  5477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e087bedee884088baf210bc009f686b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e087bedee884088baf210bc009f686b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 37453 } } }
I20240501 08:11:11.274799  5478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e087bedee884088baf210bc009f686b. Latest consensus state: current_term: 1 leader_uuid: "3e087bedee884088baf210bc009f686b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e087bedee884088baf210bc009f686b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 37453 } } }
I20240501 08:11:11.275228  5477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:11.275225  5478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:11.277887  5485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:11.284577  5485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:11.299141  5485 catalog_manager.cc:1347] Generated new cluster ID: 36e03d4341a9476d81d9e749f8fed962
I20240501 08:11:11.299257  5485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:11.330633  5485 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:11.331815  5485 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:11.346056  5485 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3e087bedee884088baf210bc009f686b: Generated new TSK 0
I20240501 08:11:11.347028  5485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:11.364305  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.129:0
--local_ip_for_outbound_sockets=127.2.248.129
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--builtin_ntp_servers=127.2.248.148:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:11:11.679636  5495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:11.679932  5495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:11.680075  5495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:11.687299  5495 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:11.687429  5495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:11.687594  5495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.129
I20240501 08:11:11.696321  5495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.129
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:11.697558  5495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:11.699203  5495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:11.711210  5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.711755  5501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.712445  5503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:11.713259  5495 server_base.cc:1034] running on GCE node
I20240501 08:11:11.713819  5495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:11.714736  5495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:11.715953  5495 hybrid_clock.cc:648] HybridClock initialized: now 1714551071715897 us; error 51 us; skew 500 ppm
I20240501 08:11:11.716305  5495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:11.718422  5495 webserver.cc:460] Webserver started at http://127.2.248.129:45669/ using document root <none> and password file <none>
I20240501 08:11:11.719086  5495 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:11.719202  5495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:11.719504  5495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:11.721987  5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8e5c8e67864047b49de6db2fc2636b61"
format_stamp: "Formatted at 2024-05-01 08:11:11 on dist-test-slave-0pdg"
I20240501 08:11:11.722712  5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8e5c8e67864047b49de6db2fc2636b61"
format_stamp: "Formatted at 2024-05-01 08:11:11 on dist-test-slave-0pdg"
I20240501 08:11:11.727633  5495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240501 08:11:11.730787  5509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:11.731945  5495 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:11.732093  5495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8e5c8e67864047b49de6db2fc2636b61"
format_stamp: "Formatted at 2024-05-01 08:11:11 on dist-test-slave-0pdg"
I20240501 08:11:11.732326  5495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:11.764281  5495 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:11.765059  5495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:11.765316  5495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:11.766186  5495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:11.767611  5495 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:11.767704  5495 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:11.767812  5495 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:11.767879  5495 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:11.797380  5495 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.129:33837
I20240501 08:11:11.797441  5621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.129:33837 every 8 connection(s)
I20240501 08:11:11.799096  5495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:11:11.808040  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 5495
I20240501 08:11:11.808265  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:11:11.811468  5622 heartbeater.cc:347] Connected to a master server at 127.2.248.190:37453
I20240501 08:11:11.811570  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.130:0
--local_ip_for_outbound_sockets=127.2.248.130
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--builtin_ntp_servers=127.2.248.148:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:11.811774  5622 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:11.812741  5622 heartbeater.cc:511] Master 127.2.248.190:37453 requested a full tablet report, sending...
I20240501 08:11:11.814797  5436 ts_manager.cc:194] Registered new tserver with Master: 8e5c8e67864047b49de6db2fc2636b61 (127.2.248.129:33837)
I20240501 08:11:11.816239  5436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.129:55345
W20240501 08:11:12.126576  5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:12.126821  5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:12.126911  5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:12.133898  5626 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:12.134006  5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:12.134126  5626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.130
I20240501 08:11:12.142711  5626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.130
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:12.143889  5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:12.145531  5626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:12.156684  5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:12.157225  5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:12.157856  5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:12.158630  5626 server_base.cc:1034] running on GCE node
I20240501 08:11:12.159188  5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:12.160055  5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:12.161283  5626 hybrid_clock.cc:648] HybridClock initialized: now 1714551072161247 us; error 39 us; skew 500 ppm
I20240501 08:11:12.161562  5626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:12.163661  5626 webserver.cc:460] Webserver started at http://127.2.248.130:38263/ using document root <none> and password file <none>
I20240501 08:11:12.164362  5626 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:12.164474  5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:12.164770  5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:12.167148  5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0761fa1241af4e649e66375ec8f630b2"
format_stamp: "Formatted at 2024-05-01 08:11:12 on dist-test-slave-0pdg"
I20240501 08:11:12.167863  5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0761fa1241af4e649e66375ec8f630b2"
format_stamp: "Formatted at 2024-05-01 08:11:12 on dist-test-slave-0pdg"
I20240501 08:11:12.173269  5626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.001s
I20240501 08:11:12.176986  5640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:12.178514  5626 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:11:12.178663  5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0761fa1241af4e649e66375ec8f630b2"
format_stamp: "Formatted at 2024-05-01 08:11:12 on dist-test-slave-0pdg"
I20240501 08:11:12.178944  5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:12.194880  5626 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:12.195608  5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:12.195847  5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:12.196751  5626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:12.198442  5626 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:12.198524  5626 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:12.198601  5626 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:12.198652  5626 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:12.228466  5626 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.130:42333
I20240501 08:11:12.228530  5752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.130:42333 every 8 connection(s)
I20240501 08:11:12.229696  5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:11:12.233297  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 5626
I20240501 08:11:12.233498  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:11:12.236663  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:0
--local_ip_for_outbound_sockets=127.2.248.131
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--builtin_ntp_servers=127.2.248.148:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:12.244215  5753 heartbeater.cc:347] Connected to a master server at 127.2.248.190:37453
I20240501 08:11:12.244499  5753 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:12.245361  5753 heartbeater.cc:511] Master 127.2.248.190:37453 requested a full tablet report, sending...
I20240501 08:11:12.246891  5436 ts_manager.cc:194] Registered new tserver with Master: 0761fa1241af4e649e66375ec8f630b2 (127.2.248.130:42333)
I20240501 08:11:12.248104  5436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.130:50031
W20240501 08:11:12.562450  5756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:12.562700  5756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:12.562831  5756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:12.569633  5756 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:12.569733  5756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:12.569881  5756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:11:12.578693  5756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:12.580090  5756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:12.581806  5756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:12.593488  5761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:12.594278  5764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:12.594756  5762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:12.595507  5756 server_base.cc:1034] running on GCE node
I20240501 08:11:12.596051  5756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:12.596983  5756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:12.598188  5756 hybrid_clock.cc:648] HybridClock initialized: now 1714551072598139 us; error 48 us; skew 500 ppm
I20240501 08:11:12.598471  5756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:12.600739  5756 webserver.cc:460] Webserver started at http://127.2.248.131:45011/ using document root <none> and password file <none>
I20240501 08:11:12.601426  5756 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:12.601536  5756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:12.601830  5756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:12.604321  5756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5de03c8390e742f89327fea09c6024ff"
format_stamp: "Formatted at 2024-05-01 08:11:12 on dist-test-slave-0pdg"
I20240501 08:11:12.605057  5756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5de03c8390e742f89327fea09c6024ff"
format_stamp: "Formatted at 2024-05-01 08:11:12 on dist-test-slave-0pdg"
I20240501 08:11:12.610205  5756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240501 08:11:12.613441  5770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:12.614645  5756 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:11:12.614801  5756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5de03c8390e742f89327fea09c6024ff"
format_stamp: "Formatted at 2024-05-01 08:11:12 on dist-test-slave-0pdg"
I20240501 08:11:12.614980  5756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:12.630482  5756 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:12.631361  5756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:12.631657  5756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:12.632670  5756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:12.634229  5756 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:12.634346  5756 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:12.634454  5756 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:12.634518  5756 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:12.664958  5756 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:36379
I20240501 08:11:12.665030  5882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:36379 every 8 connection(s)
I20240501 08:11:12.666178  5756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:12.671078  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 5756
I20240501 08:11:12.671247  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:11:12.674243  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:0
--local_ip_for_outbound_sockets=127.2.248.132
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--builtin_ntp_servers=127.2.248.148:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:12.678802  5883 heartbeater.cc:347] Connected to a master server at 127.2.248.190:37453
I20240501 08:11:12.679090  5883 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:12.679802  5883 heartbeater.cc:511] Master 127.2.248.190:37453 requested a full tablet report, sending...
I20240501 08:11:12.681046  5436 ts_manager.cc:194] Registered new tserver with Master: 5de03c8390e742f89327fea09c6024ff (127.2.248.131:36379)
I20240501 08:11:12.682009  5436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:40213
I20240501 08:11:12.818804  5622 heartbeater.cc:503] Master 127.2.248.190:37453 was elected leader, sending a full tablet report...
W20240501 08:11:12.986877  5886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:12.987135  5886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:12.987255  5886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:12.994503  5886 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:12.994606  5886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:12.994755  5886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:11:13.003320  5886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:13.004659  5886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:13.008103  5886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:13.020324  5891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:13.020840  5894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:13.021585  5892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:13.022270  5886 server_base.cc:1034] running on GCE node
I20240501 08:11:13.022850  5886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:13.023769  5886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:13.024957  5886 hybrid_clock.cc:648] HybridClock initialized: now 1714551073024912 us; error 36 us; skew 500 ppm
I20240501 08:11:13.025249  5886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:13.027410  5886 webserver.cc:460] Webserver started at http://127.2.248.132:46125/ using document root <none> and password file <none>
I20240501 08:11:13.028100  5886 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:13.028259  5886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:13.028589  5886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:13.031024  5886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "35f37005e3e5465a8deb7d9194d8cf2c"
format_stamp: "Formatted at 2024-05-01 08:11:13 on dist-test-slave-0pdg"
I20240501 08:11:13.031787  5886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "35f37005e3e5465a8deb7d9194d8cf2c"
format_stamp: "Formatted at 2024-05-01 08:11:13 on dist-test-slave-0pdg"
I20240501 08:11:13.036674  5886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.003s
I20240501 08:11:13.039920  5900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:13.041242  5886 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:11:13.041414  5886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "35f37005e3e5465a8deb7d9194d8cf2c"
format_stamp: "Formatted at 2024-05-01 08:11:13 on dist-test-slave-0pdg"
I20240501 08:11:13.041608  5886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:13.057931  5886 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:13.058729  5886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:13.059000  5886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:13.059916  5886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:13.061417  5886 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:13.061522  5886 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:13.061637  5886 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:13.061709  5886 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:13.093638  5886 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:37605
I20240501 08:11:13.093706  6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:37605 every 8 connection(s)
I20240501 08:11:13.094874  5886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:13.096212  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 5886
I20240501 08:11:13.096375  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:11:13.099399  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:0
--local_ip_for_outbound_sockets=127.2.248.133
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--builtin_ntp_servers=127.2.248.148:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:13.107355  6013 heartbeater.cc:347] Connected to a master server at 127.2.248.190:37453
I20240501 08:11:13.107731  6013 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:13.108678  6013 heartbeater.cc:511] Master 127.2.248.190:37453 requested a full tablet report, sending...
I20240501 08:11:13.110219  5436 ts_manager.cc:194] Registered new tserver with Master: 35f37005e3e5465a8deb7d9194d8cf2c (127.2.248.132:37605)
I20240501 08:11:13.111016  5436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:47039
I20240501 08:11:13.250876  5753 heartbeater.cc:503] Master 127.2.248.190:37453 was elected leader, sending a full tablet report...
W20240501 08:11:13.420992  6016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:13.421253  6016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:13.421386  6016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:13.428439  6016 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:13.428546  6016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:13.428695  6016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:11:13.437242  6016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:37453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:13.438532  6016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:13.440268  6016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:11:13.452512  6016 server_base.cc:1034] running on GCE node
W20240501 08:11:13.452919  6024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:13.453517  6021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:13.453541  6022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:13.454233  6016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:13.455121  6016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:13.456327  6016 hybrid_clock.cc:648] HybridClock initialized: now 1714551073456285 us; error 38 us; skew 500 ppm
I20240501 08:11:13.456604  6016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:13.458717  6016 webserver.cc:460] Webserver started at http://127.2.248.133:35389/ using document root <none> and password file <none>
I20240501 08:11:13.459367  6016 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:13.459477  6016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:13.459780  6016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:13.462211  6016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "19e355b84ec34fb0a5e9312beffce3e3"
format_stamp: "Formatted at 2024-05-01 08:11:13 on dist-test-slave-0pdg"
I20240501 08:11:13.462893  6016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "19e355b84ec34fb0a5e9312beffce3e3"
format_stamp: "Formatted at 2024-05-01 08:11:13 on dist-test-slave-0pdg"
I20240501 08:11:13.467489  6016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 08:11:13.470566  6030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:13.471828  6016 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:11:13.471984  6016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "19e355b84ec34fb0a5e9312beffce3e3"
format_stamp: "Formatted at 2024-05-01 08:11:13 on dist-test-slave-0pdg"
I20240501 08:11:13.472199  6016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:13.502341  6016 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:13.503171  6016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:13.503438  6016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:13.504431  6016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:13.505851  6016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:13.505945  6016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:13.506072  6016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:13.506143  6016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:13.536376  6016 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:34097
I20240501 08:11:13.536439  6142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:34097 every 8 connection(s)
I20240501 08:11:13.537618  6016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:13.541468  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 6016
I20240501 08:11:13.541633  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:11:13.549892  6143 heartbeater.cc:347] Connected to a master server at 127.2.248.190:37453
I20240501 08:11:13.550146  6143 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:13.550864  6143 heartbeater.cc:511] Master 127.2.248.190:37453 requested a full tablet report, sending...
I20240501 08:11:13.552243  5436 ts_manager.cc:194] Registered new tserver with Master: 19e355b84ec34fb0a5e9312beffce3e3 (127.2.248.133:34097)
I20240501 08:11:13.552933  5436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:36535
I20240501 08:11:13.557124  3042 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:11:13.571130  5436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37744:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240501 08:11:13.611804  6078 tablet_service.cc:1467] Processing CreateTablet for tablet 075c1b1bb9b14f1283c3db6c3da743cb (DEFAULT_TABLE table=TestTable [id=f2f85bcd4afe4c57aeff3c0eab85e317]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:13.611974  5818 tablet_service.cc:1467] Processing CreateTablet for tablet 075c1b1bb9b14f1283c3db6c3da743cb (DEFAULT_TABLE table=TestTable [id=f2f85bcd4afe4c57aeff3c0eab85e317]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:13.611974  5688 tablet_service.cc:1467] Processing CreateTablet for tablet 075c1b1bb9b14f1283c3db6c3da743cb (DEFAULT_TABLE table=TestTable [id=f2f85bcd4afe4c57aeff3c0eab85e317]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:13.613265  6078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075c1b1bb9b14f1283c3db6c3da743cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.613291  5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075c1b1bb9b14f1283c3db6c3da743cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.613312  5818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075c1b1bb9b14f1283c3db6c3da743cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.623099  6161 tablet_bootstrap.cc:492] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3: Bootstrap starting.
I20240501 08:11:13.625881  6162 tablet_bootstrap.cc:492] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff: Bootstrap starting.
I20240501 08:11:13.626336  6161 tablet_bootstrap.cc:654] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:13.627588  6161 log.cc:824] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.629547  6162 tablet_bootstrap.cc:654] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:13.630982  6162 log.cc:824] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.631733  6163 tablet_bootstrap.cc:492] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2: Bootstrap starting.
I20240501 08:11:13.633731  6162 tablet_bootstrap.cc:492] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff: No bootstrap required, opened a new log
I20240501 08:11:13.633955  6162 ts_tablet_manager.cc:1367] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240501 08:11:13.634071  6161 tablet_bootstrap.cc:492] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3: No bootstrap required, opened a new log
I20240501 08:11:13.634315  6161 ts_tablet_manager.cc:1367] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.005s
I20240501 08:11:13.635632  6163 tablet_bootstrap.cc:654] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:13.636889  6163 log.cc:824] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.639390  6163 tablet_bootstrap.cc:492] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2: No bootstrap required, opened a new log
I20240501 08:11:13.639678  6163 ts_tablet_manager.cc:1367] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240501 08:11:13.640610  6162 raft_consensus.cc:348] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.640919  6162 raft_consensus.cc:374] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:13.641036  6162 raft_consensus.cc:729] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de03c8390e742f89327fea09c6024ff, State: Initialized, Role: FOLLOWER
I20240501 08:11:13.641607  6162 consensus_queue.cc:260] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.642598  5883 heartbeater.cc:503] Master 127.2.248.190:37453 was elected leader, sending a full tablet report...
I20240501 08:11:13.642778  6161 raft_consensus.cc:348] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.643126  6162 ts_tablet_manager.cc:1398] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20240501 08:11:13.643137  6161 raft_consensus.cc:374] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:13.643222  6161 raft_consensus.cc:729] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e355b84ec34fb0a5e9312beffce3e3, State: Initialized, Role: FOLLOWER
I20240501 08:11:13.643803  6161 consensus_queue.cc:260] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.644799  6143 heartbeater.cc:503] Master 127.2.248.190:37453 was elected leader, sending a full tablet report...
I20240501 08:11:13.645413  6161 ts_tablet_manager.cc:1398] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3: Time spent starting tablet: real 0.011s	user 0.000s	sys 0.012s
I20240501 08:11:13.647795  6163 raft_consensus.cc:348] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.648243  6163 raft_consensus.cc:374] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:13.648358  6163 raft_consensus.cc:729] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0761fa1241af4e649e66375ec8f630b2, State: Initialized, Role: FOLLOWER
I20240501 08:11:13.648952  6163 consensus_queue.cc:260] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.650115  6163 ts_tablet_manager.cc:1398] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
W20240501 08:11:13.668196  5884 tablet.cc:2369] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:13.730960  6169 raft_consensus.cc:482] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:13.731354  6169 raft_consensus.cc:504] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
W20240501 08:11:13.732098  5754 tablet.cc:2369] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:13.733554  6169 leader_election.cc:288] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5de03c8390e742f89327fea09c6024ff (127.2.248.131:36379), 19e355b84ec34fb0a5e9312beffce3e3 (127.2.248.133:34097)
I20240501 08:11:13.740542  5838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075c1b1bb9b14f1283c3db6c3da743cb" candidate_uuid: "0761fa1241af4e649e66375ec8f630b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de03c8390e742f89327fea09c6024ff" is_pre_election: true
I20240501 08:11:13.740610  6098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075c1b1bb9b14f1283c3db6c3da743cb" candidate_uuid: "0761fa1241af4e649e66375ec8f630b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19e355b84ec34fb0a5e9312beffce3e3" is_pre_election: true
I20240501 08:11:13.741025  6098 raft_consensus.cc:2454] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0761fa1241af4e649e66375ec8f630b2 in term 0.
I20240501 08:11:13.741035  5838 raft_consensus.cc:2454] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0761fa1241af4e649e66375ec8f630b2 in term 0.
I20240501 08:11:13.741658  5642 leader_election.cc:302] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0761fa1241af4e649e66375ec8f630b2, 5de03c8390e742f89327fea09c6024ff; no voters: 
I20240501 08:11:13.742025  6169 raft_consensus.cc:2781] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:13.742172  6169 raft_consensus.cc:482] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:13.742269  6169 raft_consensus.cc:3037] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:13.744071  6169 raft_consensus.cc:504] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.744633  6169 leader_election.cc:288] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5de03c8390e742f89327fea09c6024ff (127.2.248.131:36379), 19e355b84ec34fb0a5e9312beffce3e3 (127.2.248.133:34097)
I20240501 08:11:13.744939  5838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075c1b1bb9b14f1283c3db6c3da743cb" candidate_uuid: "0761fa1241af4e649e66375ec8f630b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de03c8390e742f89327fea09c6024ff"
I20240501 08:11:13.745092  6098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075c1b1bb9b14f1283c3db6c3da743cb" candidate_uuid: "0761fa1241af4e649e66375ec8f630b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19e355b84ec34fb0a5e9312beffce3e3"
I20240501 08:11:13.745133  5838 raft_consensus.cc:3037] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:13.745268  6098 raft_consensus.cc:3037] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:13.747152  5838 raft_consensus.cc:2454] T 075c1b1bb9b14f1283c3db6c3da743cb P 5de03c8390e742f89327fea09c6024ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0761fa1241af4e649e66375ec8f630b2 in term 1.
I20240501 08:11:13.747300  6098 raft_consensus.cc:2454] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0761fa1241af4e649e66375ec8f630b2 in term 1.
I20240501 08:11:13.747498  5642 leader_election.cc:302] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0761fa1241af4e649e66375ec8f630b2, 5de03c8390e742f89327fea09c6024ff; no voters: 
I20240501 08:11:13.747736  6169 raft_consensus.cc:2781] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:13.748142  6169 raft_consensus.cc:686] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [term 1 LEADER]: Becoming Leader. State: Replica: 0761fa1241af4e649e66375ec8f630b2, State: Running, Role: LEADER
I20240501 08:11:13.748632  6169 consensus_queue.cc:237] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } }
I20240501 08:11:13.752717  5436 catalog_manager.cc:5484] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0761fa1241af4e649e66375ec8f630b2 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "0761fa1241af4e649e66375ec8f630b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0761fa1241af4e649e66375ec8f630b2" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 42333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:13.783069  3042 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:11:13.785316  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8e5c8e67864047b49de6db2fc2636b61 to finish bootstrapping
W20240501 08:11:13.789180  6144 tablet.cc:2369] T 075c1b1bb9b14f1283c3db6c3da743cb P 19e355b84ec34fb0a5e9312beffce3e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:13.793947  3042 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0761fa1241af4e649e66375ec8f630b2 to finish bootstrapping
I20240501 08:11:13.801322  3042 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5de03c8390e742f89327fea09c6024ff to finish bootstrapping
I20240501 08:11:13.808068  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 35f37005e3e5465a8deb7d9194d8cf2c to finish bootstrapping
I20240501 08:11:13.816011  3042 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 19e355b84ec34fb0a5e9312beffce3e3 to finish bootstrapping
I20240501 08:11:14.113461  6013 heartbeater.cc:503] Master 127.2.248.190:37453 was elected leader, sending a full tablet report...
I20240501 08:11:14.140163  6169 consensus_queue.cc:1035] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de03c8390e742f89327fea09c6024ff" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:14.148178  6172 consensus_queue.cc:1035] T 075c1b1bb9b14f1283c3db6c3da743cb P 0761fa1241af4e649e66375ec8f630b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19e355b84ec34fb0a5e9312beffce3e3" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:14.267288  6078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:14.269166  5688 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:14.274609  5818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:14.275602  5948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:14.289068  5557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:14.430723  5688 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:14.438418  6078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:14.439374  5948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:14.447088  5818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:14.448956  5557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.027805  6078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.029655  5948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.032263  5688 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:15.033094  5818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.037216  5557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.189646  6078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.191406  5688 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:15.202469  5948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.213017  5557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.214241  5818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.370604  5688 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:15.374912  6078 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.377938  5948 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.379480  5557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.392859  5818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.535619  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 5495
I20240501 08:11:15.537118  5617 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:15.629005  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 5495
I20240501 08:11:15.646570  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 5626
I20240501 08:11:15.647506  5748 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:15.746695  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 5626
W20240501 08:11:15.763320  5772 connection.cc:535] server connection from 127.2.248.130:54969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:15.763558  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 5756
I20240501 08:11:15.764719  5877 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:15.854549  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 5756
I20240501 08:11:15.871816  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 5886
I20240501 08:11:15.872926  6008 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:15.958547  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 5886
I20240501 08:11:15.974491  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 6016
I20240501 08:11:15.975454  6137 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:16.068524  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 6016
I20240501 08:11:16.085592  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 5405
I20240501 08:11:16.086726  5466 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:16.197768  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 5405
2024-05-01T08:11:16Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5435 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5435 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/3
I20240501 08:11:16.240303  3042 test_util.cc:268] Using random seed: 131677076
I20240501 08:11:16.242105  3042 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:16.242180  3042 ts_itest-base.cc:113] --------------
I20240501 08:11:16.242250  3042 ts_itest-base.cc:114] 7 tablet servers
I20240501 08:11:16.242298  3042 ts_itest-base.cc:115] 2 replicas per TS
I20240501 08:11:16.242363  3042 ts_itest-base.cc:116] --------------
2024-05-01T08:11:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:16Z Disabled control of system clock
I20240501 08:11:16.273299  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:39497
--webserver_interface=127.2.248.190
--webserver_port=0
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.248.190:39497
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 08:11:16.572707  6370 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 08:11:16.573000  6370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:16.573115  6370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:16.573212  6370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:16.579769  6370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:16.579857  6370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:16.579929  6370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:16.579991  6370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:16.580077  6370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:16.588260  6370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.248.190:39497
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:39497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.248.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:16.589401  6370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:16.591012  6370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:16.601375  6375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:16.601833  6378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:16.602556  6376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:16.603263  6370 server_base.cc:1034] running on GCE node
I20240501 08:11:16.603961  6370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:16.605201  6370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:16.606417  6370 hybrid_clock.cc:648] HybridClock initialized: now 1714551076606352 us; error 46 us; skew 500 ppm
I20240501 08:11:16.606707  6370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:16.608848  6370 webserver.cc:460] Webserver started at http://127.2.248.190:39907/ using document root <none> and password file <none>
I20240501 08:11:16.609552  6370 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:16.609679  6370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:16.609943  6370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:16.612459  6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "42d8139962d445eba456f56990374af2"
format_stamp: "Formatted at 2024-05-01 08:11:16 on dist-test-slave-0pdg"
I20240501 08:11:16.613129  6370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "42d8139962d445eba456f56990374af2"
format_stamp: "Formatted at 2024-05-01 08:11:16 on dist-test-slave-0pdg"
I20240501 08:11:16.617617  6370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 08:11:16.620579  6384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:16.621742  6370 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20240501 08:11:16.621891  6370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
uuid: "42d8139962d445eba456f56990374af2"
format_stamp: "Formatted at 2024-05-01 08:11:16 on dist-test-slave-0pdg"
I20240501 08:11:16.622045  6370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:16.637893  6370 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:16.638586  6370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:16.638842  6370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:16.655794  6370 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.190:39497
I20240501 08:11:16.655823  6435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.190:39497 every 8 connection(s)
I20240501 08:11:16.656997  6370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:11:16.660388  6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:16.661003  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 6370
I20240501 08:11:16.661195  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:11:16.671135  6436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2: Bootstrap starting.
I20240501 08:11:16.674604  6436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:16.675437  6436 log.cc:824] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:16.677525  6436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2: No bootstrap required, opened a new log
I20240501 08:11:16.682902  6436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d8139962d445eba456f56990374af2" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 39497 } }
I20240501 08:11:16.683184  6436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:16.683270  6436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d8139962d445eba456f56990374af2, State: Initialized, Role: FOLLOWER
I20240501 08:11:16.683760  6436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d8139962d445eba456f56990374af2" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 39497 } }
I20240501 08:11:16.683934  6436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:16.684062  6436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:16.684221  6436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:16.686072  6436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d8139962d445eba456f56990374af2" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 39497 } }
I20240501 08:11:16.686492  6436 leader_election.cc:302] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d8139962d445eba456f56990374af2; no voters: 
I20240501 08:11:16.686976  6436 leader_election.cc:288] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:11:16.687079  6441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:16.687692  6441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [term 1 LEADER]: Becoming Leader. State: Replica: 42d8139962d445eba456f56990374af2, State: Running, Role: LEADER
I20240501 08:11:16.688386  6441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d8139962d445eba456f56990374af2" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 39497 } }
I20240501 08:11:16.688781  6436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:16.691227  6442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42d8139962d445eba456f56990374af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d8139962d445eba456f56990374af2" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 39497 } } }
I20240501 08:11:16.691550  6442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:16.691251  6443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42d8139962d445eba456f56990374af2. Latest consensus state: current_term: 1 leader_uuid: "42d8139962d445eba456f56990374af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d8139962d445eba456f56990374af2" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 39497 } } }
I20240501 08:11:16.691763  6443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:16.697062  6446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:16.700733  6446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:16.707774  6446 catalog_manager.cc:1347] Generated new cluster ID: 57c0b2d2b8aa494bbfde81a2a78f4edd
I20240501 08:11:16.707867  6446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:16.715514  6446 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:16.716547  6446 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:16.727006  6446 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 42d8139962d445eba456f56990374af2: Generated new TSK 0
I20240501 08:11:16.727769  6446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:16.738322  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.129:0
--local_ip_for_outbound_sockets=127.2.248.129
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:11:17.049211  6460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:17.049482  6460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:17.049566  6460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:17.056450  6460 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:17.056547  6460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:17.056650  6460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.129
I20240501 08:11:17.064854  6460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.129
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:17.066002  6460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:17.067520  6460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:17.079028  6465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:17.079824  6460 server_base.cc:1034] running on GCE node
W20240501 08:11:17.080432  6466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:17.080982  6468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:17.081578  6460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:17.082401  6460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:17.083597  6460 hybrid_clock.cc:648] HybridClock initialized: now 1714551077083560 us; error 30 us; skew 500 ppm
I20240501 08:11:17.083839  6460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:17.085978  6460 webserver.cc:460] Webserver started at http://127.2.248.129:38423/ using document root <none> and password file <none>
I20240501 08:11:17.086598  6460 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:17.086683  6460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:17.086922  6460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:17.090879  6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b0190f14862148f4bde78afbbf1ba43d"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.091576  6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b0190f14862148f4bde78afbbf1ba43d"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.096453  6460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240501 08:11:17.099372  6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:17.100587  6460 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
I20240501 08:11:17.100706  6460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b0190f14862148f4bde78afbbf1ba43d"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.100854  6460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:17.124858  6460 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:17.125594  6460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:17.125835  6460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:17.126694  6460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:17.128018  6460 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:17.128127  6460 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:17.128263  6460 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:17.128335  6460 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:17.159904  6460 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.129:41709
I20240501 08:11:17.159999  6586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.129:41709 every 8 connection(s)
I20240501 08:11:17.161218  6460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:11:17.170529  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 6460
I20240501 08:11:17.170732  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:11:17.173413  6587 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:17.173683  6587 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:17.174100  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.130:0
--local_ip_for_outbound_sockets=127.2.248.130
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:17.174481  6587 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:17.176447  6401 ts_manager.cc:194] Registered new tserver with Master: b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:17.177922  6401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.129:37345
W20240501 08:11:17.499727  6590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:17.500000  6590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:17.500077  6590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:17.507229  6590 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:17.507342  6590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:17.507454  6590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.130
I20240501 08:11:17.516044  6590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.130
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:17.517314  6590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:17.518983  6590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:17.530995  6598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:17.531539  6595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:17.532228  6596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:17.532943  6590 server_base.cc:1034] running on GCE node
I20240501 08:11:17.533469  6590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:17.534384  6590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:17.535609  6590 hybrid_clock.cc:648] HybridClock initialized: now 1714551077535558 us; error 52 us; skew 500 ppm
I20240501 08:11:17.535902  6590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:17.538081  6590 webserver.cc:460] Webserver started at http://127.2.248.130:38787/ using document root <none> and password file <none>
I20240501 08:11:17.538765  6590 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:17.538877  6590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:17.539163  6590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:17.541652  6590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d143a9ab17334a6081de63802bdce654"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.542392  6590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d143a9ab17334a6081de63802bdce654"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.547210  6590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20240501 08:11:17.550240  6604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:17.551487  6590 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:17.551630  6590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d143a9ab17334a6081de63802bdce654"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.551802  6590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:17.563486  6590 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:17.564224  6590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:17.564473  6590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:17.565300  6590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:17.566598  6590 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:17.566690  6590 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:17.566799  6590 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:17.566866  6590 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:17.597579  6590 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.130:45487
I20240501 08:11:17.597612  6716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.130:45487 every 8 connection(s)
I20240501 08:11:17.598850  6590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:11:17.605553  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 6590
I20240501 08:11:17.605712  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:11:17.608685  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:0
--local_ip_for_outbound_sockets=127.2.248.131
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:17.610669  6717 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:17.610920  6717 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:17.611632  6717 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:17.613029  6401 ts_manager.cc:194] Registered new tserver with Master: d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:17.614058  6401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.130:47347
W20240501 08:11:17.928167  6720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:17.928447  6720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:17.928588  6720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:17.935789  6720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:17.935930  6720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:17.936089  6720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:11:17.944715  6720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:17.945960  6720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:17.947664  6720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:17.959457  6726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:17.960011  6728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:17.960594  6725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:17.961370  6720 server_base.cc:1034] running on GCE node
I20240501 08:11:17.961953  6720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:17.962864  6720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:17.964087  6720 hybrid_clock.cc:648] HybridClock initialized: now 1714551077964032 us; error 44 us; skew 500 ppm
I20240501 08:11:17.964458  6720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:17.966485  6720 webserver.cc:460] Webserver started at http://127.2.248.131:39237/ using document root <none> and password file <none>
I20240501 08:11:17.967161  6720 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:17.967268  6720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:17.967550  6720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:17.969972  6720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7fb652773c524f639dba8c189b7743ba"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.970680  6720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7fb652773c524f639dba8c189b7743ba"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.975241  6720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 08:11:17.978255  6734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:17.979378  6720 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:11:17.979513  6720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7fb652773c524f639dba8c189b7743ba"
format_stamp: "Formatted at 2024-05-01 08:11:17 on dist-test-slave-0pdg"
I20240501 08:11:17.979687  6720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:18.004909  6720 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:18.005795  6720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:18.006094  6720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:18.007092  6720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:18.008689  6720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:18.008793  6720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.008929  6720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:18.009024  6720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.042855  6720 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:43453
I20240501 08:11:18.042901  6846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:43453 every 8 connection(s)
I20240501 08:11:18.044253  6720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:18.050285  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 6720
I20240501 08:11:18.050477  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:11:18.054021  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:0
--local_ip_for_outbound_sockets=127.2.248.132
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:18.058166  6847 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:18.058593  6847 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:18.059691  6847 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:18.061460  6401 ts_manager.cc:194] Registered new tserver with Master: 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:18.062574  6401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:38041
I20240501 08:11:18.180467  6587 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
W20240501 08:11:18.381633  6850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:18.381966  6850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:18.382130  6850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:18.389582  6850 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:18.389726  6850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:18.389885  6850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:11:18.398777  6850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:18.400105  6850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:18.401885  6850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:18.414491  6856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:18.415009  6855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:18.415668  6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:18.416512  6850 server_base.cc:1034] running on GCE node
I20240501 08:11:18.417081  6850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:18.418004  6850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:18.419216  6850 hybrid_clock.cc:648] HybridClock initialized: now 1714551078419181 us; error 47 us; skew 500 ppm
I20240501 08:11:18.419536  6850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:18.421662  6850 webserver.cc:460] Webserver started at http://127.2.248.132:43137/ using document root <none> and password file <none>
I20240501 08:11:18.422389  6850 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:18.422502  6850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:18.422788  6850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:18.425259  6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fcfccbafc3254a87896f0e72c8078ab5"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:18.426015  6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fcfccbafc3254a87896f0e72c8078ab5"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:18.430867  6850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240501 08:11:18.434121  6864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.435333  6850 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:18.435487  6850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fcfccbafc3254a87896f0e72c8078ab5"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:18.435667  6850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:18.449975  6850 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:18.450736  6850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:18.450985  6850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:18.451871  6850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:18.453272  6850 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:18.453364  6850 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.453478  6850 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:18.453547  6850 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.481930  6850 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:36129
I20240501 08:11:18.481976  6976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:36129 every 8 connection(s)
I20240501 08:11:18.483209  6850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:18.487582  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 6850
I20240501 08:11:18.487736  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:11:18.490855  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:0
--local_ip_for_outbound_sockets=127.2.248.133
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:18.495852  6977 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:18.496168  6977 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:18.496917  6977 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:18.498235  6401 ts_manager.cc:194] Registered new tserver with Master: fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129)
I20240501 08:11:18.499323  6401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:34237
I20240501 08:11:18.616341  6717 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
W20240501 08:11:18.811192  6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:18.811460  6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:18.811578  6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:18.818670  6980 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:18.818781  6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:18.818935  6980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:11:18.827809  6980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:18.829102  6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:18.830770  6980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:18.842537  6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:18.843050  6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:18.843689  6986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:18.844558  6980 server_base.cc:1034] running on GCE node
I20240501 08:11:18.845150  6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:18.846493  6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:18.847738  6980 hybrid_clock.cc:648] HybridClock initialized: now 1714551078847669 us; error 54 us; skew 500 ppm
I20240501 08:11:18.848054  6980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:18.850598  6980 webserver.cc:460] Webserver started at http://127.2.248.133:40057/ using document root <none> and password file <none>
I20240501 08:11:18.851414  6980 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:18.851516  6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:18.851775  6980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:18.854405  6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0257c0e8038d4e3fa9580cb47278f16f"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:18.855136  6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0257c0e8038d4e3fa9580cb47278f16f"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:18.860020  6980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240501 08:11:18.863327  6994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.864647  6980 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240501 08:11:18.864789  6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0257c0e8038d4e3fa9580cb47278f16f"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:18.864964  6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:18.878261  6980 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:18.879125  6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:18.879386  6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:18.880391  6980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:18.881896  6980 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:18.881979  6980 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.882089  6980 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:18.882171  6980 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:18.912938  6980 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:43177
I20240501 08:11:18.912987  7107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:43177 every 8 connection(s)
I20240501 08:11:18.914273  6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:18.914557  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 6980
I20240501 08:11:18.914741  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:11:18.918255  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.134:0
--local_ip_for_outbound_sockets=127.2.248.134
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:18.928642  7108 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:18.928910  7108 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:18.929634  7108 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:18.930999  6401 ts_manager.cc:194] Registered new tserver with Master: 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177)
I20240501 08:11:18.932181  6401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:37061
I20240501 08:11:19.064967  6847 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
W20240501 08:11:19.239030  7111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:19.239277  7111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:19.239387  7111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:19.246376  7111 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:19.246480  7111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:19.246585  7111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.134
I20240501 08:11:19.254940  7111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.134
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:19.256108  7111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:19.257812  7111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:19.269089  7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:19.269676  7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:19.270370  7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:19.271126  7111 server_base.cc:1034] running on GCE node
I20240501 08:11:19.271670  7111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:19.272603  7111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:19.273800  7111 hybrid_clock.cc:648] HybridClock initialized: now 1714551079273763 us; error 27 us; skew 500 ppm
I20240501 08:11:19.274078  7111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:19.276185  7111 webserver.cc:460] Webserver started at http://127.2.248.134:37369/ using document root <none> and password file <none>
I20240501 08:11:19.276836  7111 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:19.276940  7111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:19.277251  7111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:19.279675  7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c284970db26b4c95a9ee09310b5adcbe"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:19.280496  7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c284970db26b4c95a9ee09310b5adcbe"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:19.285208  7111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 08:11:19.288420  7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:19.289752  7111 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:11:19.289889  7111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c284970db26b4c95a9ee09310b5adcbe"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:19.290068  7111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:19.318051  7111 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:19.318817  7111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:19.319072  7111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:19.319939  7111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:19.321362  7111 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:19.321455  7111 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:19.321560  7111 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:19.321628  7111 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:19.352694  7111 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.134:33609
I20240501 08:11:19.352758  7237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.134:33609 every 8 connection(s)
I20240501 08:11:19.353945  7111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:19.361742  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 7111
I20240501 08:11:19.361932  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:11:19.365322  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.135:0
--local_ip_for_outbound_sockets=127.2.248.135
--webserver_interface=127.2.248.135
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:19.366094  7238 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:19.366377  7238 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:19.367087  7238 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:19.368464  6401 ts_manager.cc:194] Registered new tserver with Master: c284970db26b4c95a9ee09310b5adcbe (127.2.248.134:33609)
I20240501 08:11:19.369239  6401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.134:39281
I20240501 08:11:19.502274  6977 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
W20240501 08:11:19.677476  7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:19.677714  7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:19.677841  7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:19.685004  7241 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:19.685132  7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:19.685276  7241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.135
I20240501 08:11:19.693580  7241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.248.135
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.135
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:19.694800  7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:19.696432  7241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:19.708187  7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:19.708659  7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:19.709295  7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:19.710062  7241 server_base.cc:1034] running on GCE node
I20240501 08:11:19.710578  7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:19.711424  7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:19.712636  7241 hybrid_clock.cc:648] HybridClock initialized: now 1714551079712590 us; error 40 us; skew 500 ppm
I20240501 08:11:19.712939  7241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:19.715119  7241 webserver.cc:460] Webserver started at http://127.2.248.135:40711/ using document root <none> and password file <none>
I20240501 08:11:19.715791  7241 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:19.715899  7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:19.716238  7241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:19.718634  7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ad679e52ba5643f2b11838d14696d1ed"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:19.719347  7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ad679e52ba5643f2b11838d14696d1ed"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:19.724185  7241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240501 08:11:19.727279  7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:19.728479  7241 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:11:19.728631  7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ad679e52ba5643f2b11838d14696d1ed"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:19.728811  7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:19.752548  7241 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:19.753326  7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:19.753582  7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:19.754451  7241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:19.755833  7241 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:19.755935  7241 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:19.756042  7241 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:19.756129  7241 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:19.787259  7241 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.135:36997
I20240501 08:11:19.787307  7367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.135:36997 every 8 connection(s)
I20240501 08:11:19.788695  7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:19.795699  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 7241
I20240501 08:11:19.795894  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:11:19.801316  7368 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:19.801556  7368 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:19.802248  7368 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:19.803445  6401 ts_manager.cc:194] Registered new tserver with Master: ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997)
I20240501 08:11:19.804239  6401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.135:59537
I20240501 08:11:19.811621  3042 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:11:19.825990  6401 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34732:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240501 08:11:19.865187  6522 tablet_service.cc:1467] Processing CreateTablet for tablet b421e60e689c4adfabadb47bd9a9e167 (DEFAULT_TABLE table=TestTable [id=81f06f1786d74ce48c8a6680e52f03ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:19.866025  7043 tablet_service.cc:1467] Processing CreateTablet for tablet b421e60e689c4adfabadb47bd9a9e167 (DEFAULT_TABLE table=TestTable [id=81f06f1786d74ce48c8a6680e52f03ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:19.866668  6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b421e60e689c4adfabadb47bd9a9e167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:19.867300  7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b421e60e689c4adfabadb47bd9a9e167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:19.880735  7384 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:19.880904  7385 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap starting.
I20240501 08:11:19.884490  7384 tablet_bootstrap.cc:654] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:19.884490  7385 tablet_bootstrap.cc:654] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:19.885788  7385 log.cc:824] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:19.885788  7384 log.cc:824] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:19.888635  7385 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: No bootstrap required, opened a new log
I20240501 08:11:19.888918  7384 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:19.888917  7385 ts_tablet_manager.cc:1367] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.001s
I20240501 08:11:19.889154  7384 ts_tablet_manager.cc:1367] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240501 08:11:19.897845  7384 raft_consensus.cc:348] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:19.897845  7385 raft_consensus.cc:348] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:19.898272  7385 raft_consensus.cc:374] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:19.898272  7384 raft_consensus.cc:374] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:19.898408  7384 raft_consensus.cc:729] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:19.898408  7385 raft_consensus.cc:729] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0257c0e8038d4e3fa9580cb47278f16f, State: Initialized, Role: FOLLOWER
I20240501 08:11:19.899150  7385 consensus_queue.cc:260] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:19.899149  7384 consensus_queue.cc:260] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:19.900315  7108 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
I20240501 08:11:19.901330  7384 ts_tablet_manager.cc:1398] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.004s
I20240501 08:11:19.902709  7385 ts_tablet_manager.cc:1398] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.003s
W20240501 08:11:19.914621  6588 tablet.cc:2369] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:19.916414  7109 tablet.cc:2369] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:20.065985  7389 raft_consensus.cc:482] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:20.066247  7389 raft_consensus.cc:504] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:20.067896  7389 leader_election.cc:288] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177)
I20240501 08:11:20.074093  7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421e60e689c4adfabadb47bd9a9e167" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0257c0e8038d4e3fa9580cb47278f16f" is_pre_election: true
I20240501 08:11:20.074532  7063 raft_consensus.cc:2454] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:20.075062  6478 leader_election.cc:302] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0257c0e8038d4e3fa9580cb47278f16f, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:20.075453  7389 raft_consensus.cc:2781] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:20.075577  7389 raft_consensus.cc:482] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:20.075680  7389 raft_consensus.cc:3037] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:20.077922  7389 raft_consensus.cc:504] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:20.078375  7389 leader_election.cc:288] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177)
I20240501 08:11:20.078814  7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421e60e689c4adfabadb47bd9a9e167" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0257c0e8038d4e3fa9580cb47278f16f"
I20240501 08:11:20.078963  7063 raft_consensus.cc:3037] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:20.080971  7063 raft_consensus.cc:2454] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:20.081305  6478 leader_election.cc:302] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0257c0e8038d4e3fa9580cb47278f16f, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:20.081578  7389 raft_consensus.cc:2781] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:20.081964  7389 raft_consensus.cc:686] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:20.082441  7389 consensus_queue.cc:237] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:20.086975  6399 catalog_manager.cc:5484] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:20.140352  3042 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:11:20.142335  3042 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b0190f14862148f4bde78afbbf1ba43d to finish bootstrapping
I20240501 08:11:20.155746  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d143a9ab17334a6081de63802bdce654 to finish bootstrapping
I20240501 08:11:20.165222  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7fb652773c524f639dba8c189b7743ba to finish bootstrapping
I20240501 08:11:20.175807  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fcfccbafc3254a87896f0e72c8078ab5 to finish bootstrapping
I20240501 08:11:20.184708  3042 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0257c0e8038d4e3fa9580cb47278f16f to finish bootstrapping
I20240501 08:11:20.191943  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c284970db26b4c95a9ee09310b5adcbe to finish bootstrapping
I20240501 08:11:20.200240  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ad679e52ba5643f2b11838d14696d1ed to finish bootstrapping
I20240501 08:11:20.208643  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 6850
I20240501 08:11:20.209579  6972 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:20.289309  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 6850
I20240501 08:11:20.305574  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 6980
I20240501 08:11:20.306558  7103 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:20.371448  7238 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
I20240501 08:11:20.391256  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 6980
I20240501 08:11:20.407925  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 7111
I20240501 08:11:20.409089  7233 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:20.488269  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 7111
I20240501 08:11:20.503811  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 7241
I20240501 08:11:20.504926  7363 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:20.583947  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 7241
W20240501 08:11:20.656184  6478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111)
W20240501 08:11:20.657632  6478 consensus_peers.cc:485] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:23.140923  6478 consensus_peers.cc:485] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:11:24.351799  6399 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34732:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:11:24.363801  6522 tablet_service.cc:1467] Processing CreateTablet for tablet 4f476973a1f24e09b30075aff39ce31c (DEFAULT_TABLE table=rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:24.364344  6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f476973a1f24e09b30075aff39ce31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:24.374560  6782 tablet_service.cc:1467] Processing CreateTablet for tablet a3beb371e6b749ec9580bca95f0908d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:24.374282  6651 tablet_service.cc:1467] Processing CreateTablet for tablet 5924bdc199d04ba495059ea3c60f2792 (DEFAULT_TABLE table=rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:24.374217  6650 tablet_service.cc:1467] Processing CreateTablet for tablet 4f476973a1f24e09b30075aff39ce31c (DEFAULT_TABLE table=rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:24.374990  7424 tablet_bootstrap.cc:492] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:24.374250  6652 tablet_service.cc:1467] Processing CreateTablet for tablet a3beb371e6b749ec9580bca95f0908d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:24.375864  6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3beb371e6b749ec9580bca95f0908d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:24.375877  6651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5924bdc199d04ba495059ea3c60f2792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:24.376367  6650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f476973a1f24e09b30075aff39ce31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:24.374200  6781 tablet_service.cc:1467] Processing CreateTablet for tablet 5924bdc199d04ba495059ea3c60f2792 (DEFAULT_TABLE table=rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:24.377079  6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5924bdc199d04ba495059ea3c60f2792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:24.377079  6652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3beb371e6b749ec9580bca95f0908d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:24.379294  7424 tablet_bootstrap.cc:654] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:24.390586  7424 tablet_bootstrap.cc:492] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:24.390851  7424 ts_tablet_manager.cc:1367] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.005s
I20240501 08:11:24.392066  7424 raft_consensus.cc:348] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:24.392235  7426 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:24.392385  7424 raft_consensus.cc:374] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:24.392508  7424 raft_consensus.cc:729] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:24.392836  7424 consensus_queue.cc:260] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:24.393548  7424 ts_tablet_manager.cc:1398] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:11:24.393741  7427 tablet_bootstrap.cc:492] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:24.397323  7426 tablet_bootstrap.cc:654] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:24.398089  7427 tablet_bootstrap.cc:654] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:24.398823  7426 log.cc:824] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:24.399101  7427 log.cc:824] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:24.401151  7426 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:24.401247  7427 tablet_bootstrap.cc:492] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:24.401432  7426 ts_tablet_manager.cc:1367] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.000s
I20240501 08:11:24.401531  7427 ts_tablet_manager.cc:1367] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.006s
I20240501 08:11:24.407881  7427 raft_consensus.cc:348] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.407874  7426 raft_consensus.cc:348] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.408231  7426 raft_consensus.cc:374] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:24.408228  7427 raft_consensus.cc:374] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:24.408322  7426 raft_consensus.cc:729] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:24.408324  7427 raft_consensus.cc:729] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:24.408905  7427 consensus_queue.cc:260] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.408905  7426 consensus_queue.cc:260] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.410498  7427 ts_tablet_manager.cc:1398] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.006s
I20240501 08:11:24.410831  7427 tablet_bootstrap.cc:492] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:24.412309  7427 tablet_bootstrap.cc:654] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:24.412302  7426 ts_tablet_manager.cc:1398] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.003s
I20240501 08:11:24.412695  7426 tablet_bootstrap.cc:492] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:24.414330  7426 tablet_bootstrap.cc:654] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:24.414613  7427 tablet_bootstrap.cc:492] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:24.414834  7427 ts_tablet_manager.cc:1367] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:11:24.415845  7427 raft_consensus.cc:348] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:24.416003  7427 raft_consensus.cc:374] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:24.416086  7427 raft_consensus.cc:729] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:24.416333  7427 consensus_queue.cc:260] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:24.416729  7427 ts_tablet_manager.cc:1398] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:24.416999  7427 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:24.417431  7426 tablet_bootstrap.cc:492] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:24.417600  7426 ts_tablet_manager.cc:1367] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.003s
I20240501 08:11:24.418473  7426 raft_consensus.cc:348] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.418650  7427 tablet_bootstrap.cc:654] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:24.418651  7426 raft_consensus.cc:374] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:24.418752  7426 raft_consensus.cc:729] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:24.418995  7426 consensus_queue.cc:260] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.419358  7426 ts_tablet_manager.cc:1398] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:11:24.421298  7413 raft_consensus.cc:482] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:24.421504  7427 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:24.421447  7413 raft_consensus.cc:504] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:24.421653  7427 ts_tablet_manager.cc:1367] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240501 08:11:24.422052  7413 leader_election.cc:288] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:24.422255  7427 raft_consensus.cc:348] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.422423  7427 raft_consensus.cc:374] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:24.422499  7427 raft_consensus.cc:729] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:24.422749  7427 consensus_queue.cc:260] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.423390  7427 ts_tablet_manager.cc:1398] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240501 08:11:24.429015  6672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f476973a1f24e09b30075aff39ce31c" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654" is_pre_election: true
I20240501 08:11:24.429488  6672 raft_consensus.cc:2454] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:24.430049  6476 leader_election.cc:302] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:24.430326  7413 raft_consensus.cc:2781] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:24.430455  7413 raft_consensus.cc:482] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:24.430553  7413 raft_consensus.cc:3037] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:24.432476  7413 raft_consensus.cc:504] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:24.432904  7413 leader_election.cc:288] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:24.433367  6672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f476973a1f24e09b30075aff39ce31c" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:24.433552  6672 raft_consensus.cc:3037] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:24.435485  6672 raft_consensus.cc:2454] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:24.435819  6476 leader_election.cc:302] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:24.436062  7413 raft_consensus.cc:2781] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:24.436463  7413 raft_consensus.cc:686] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:24.436703  7413 consensus_queue.cc:237] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:24.438720  6401 catalog_manager.cc:5484] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:24.499516  7430 raft_consensus.cc:482] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:24.499702  7430 raft_consensus.cc:504] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.501180  7430 leader_election.cc:288] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:24.507375  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3beb371e6b749ec9580bca95f0908d1" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba" is_pre_election: true
I20240501 08:11:24.507845  6802 raft_consensus.cc:2454] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 0.
I20240501 08:11:24.508431  6606 leader_election.cc:302] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:24.508783  7430 raft_consensus.cc:2781] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:24.508905  7430 raft_consensus.cc:482] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:24.509011  7430 raft_consensus.cc:3037] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:24.510931  7430 raft_consensus.cc:504] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.511395  7430 leader_election.cc:288] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Requested vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:24.511837  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3beb371e6b749ec9580bca95f0908d1" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba"
I20240501 08:11:24.512007  6802 raft_consensus.cc:3037] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:24.514075  6802 raft_consensus.cc:2454] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 1.
I20240501 08:11:24.514479  6606 leader_election.cc:302] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:24.514758  7430 raft_consensus.cc:2781] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:24.515146  7430 raft_consensus.cc:686] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Becoming Leader. State: Replica: d143a9ab17334a6081de63802bdce654, State: Running, Role: LEADER
I20240501 08:11:24.515630  7430 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.519635  6401 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 reported cstate change: term changed from 0 to 1, leader changed from <none> to d143a9ab17334a6081de63802bdce654 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:24.550558  6848 tablet.cc:2369] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:24.558229  7431 raft_consensus.cc:482] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:24.558403  7431 raft_consensus.cc:504] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.559789  7431 leader_election.cc:288] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:24.565784  6672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5924bdc199d04ba495059ea3c60f2792" candidate_uuid: "7fb652773c524f639dba8c189b7743ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654" is_pre_election: true
I20240501 08:11:24.566015  6672 raft_consensus.cc:2454] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb652773c524f639dba8c189b7743ba in term 0.
I20240501 08:11:24.566593  6736 leader_election.cc:302] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:24.566972  7431 raft_consensus.cc:2781] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:24.567101  7431 raft_consensus.cc:482] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:24.567199  7431 raft_consensus.cc:3037] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:24.569077  7431 raft_consensus.cc:504] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.569504  7431 leader_election.cc:288] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:24.569978  6672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5924bdc199d04ba495059ea3c60f2792" candidate_uuid: "7fb652773c524f639dba8c189b7743ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:24.570160  6672 raft_consensus.cc:3037] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:24.571902  6672 raft_consensus.cc:2454] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb652773c524f639dba8c189b7743ba in term 1.
I20240501 08:11:24.572258  6736 leader_election.cc:302] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:24.572522  7431 raft_consensus.cc:2781] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:24.572909  7431 raft_consensus.cc:686] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Becoming Leader. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Running, Role: LEADER
I20240501 08:11:24.573316  7431 consensus_queue.cc:237] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:24.577265  6401 catalog_manager.cc:5484] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb652773c524f639dba8c189b7743ba (127.2.248.131). New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } }
W20240501 08:11:24.605316  6718 tablet.cc:2369] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:24.895663  7413 consensus_queue.cc:1035] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:24.927597  7443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:24.927949  7443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:24.935245  7443 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:11:25.015633  7430 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:25.074018  7431 consensus_queue.cc:1035] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 219.086 ms
  time per row: 0.109543 ms
I20240501 08:11:25.294806  6398 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34732:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:11:25.303022  6781 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc900308d554b08b5e54974c0ae77e9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:25.303535  6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc900308d554b08b5e54974c0ae77e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:25.304013  6522 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc900308d554b08b5e54974c0ae77e9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:25.304360  6782 tablet_service.cc:1467] Processing CreateTablet for tablet dff918bc67204080bf1c272c2d950045 (DEFAULT_TABLE table=rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:25.304499  6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc900308d554b08b5e54974c0ae77e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:25.304602  6520 tablet_service.cc:1467] Processing CreateTablet for tablet 366e736f8763465299980fe09c0159bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:25.304713  6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff918bc67204080bf1c272c2d950045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:25.305013  6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366e736f8763465299980fe09c0159bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:25.305418  6521 tablet_service.cc:1467] Processing CreateTablet for tablet dff918bc67204080bf1c272c2d950045 (DEFAULT_TABLE table=rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:25.305929  6521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff918bc67204080bf1c272c2d950045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:25.311504  6652 tablet_service.cc:1467] Processing CreateTablet for tablet 366e736f8763465299980fe09c0159bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:25.311944  6652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366e736f8763465299980fe09c0159bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:25.313649  7505 tablet_bootstrap.cc:492] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:25.316480  7505 tablet_bootstrap.cc:654] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:25.318925  7504 tablet_bootstrap.cc:492] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:25.319836  7505 tablet_bootstrap.cc:492] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:25.319991  7507 tablet_bootstrap.cc:492] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:25.320031  7505 ts_tablet_manager.cc:1367] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.006s	user 0.006s	sys 0.000s
I20240501 08:11:25.320822  7504 tablet_bootstrap.cc:654] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:25.320854  7505 raft_consensus.cc:348] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.321060  7505 raft_consensus.cc:374] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:25.321190  7505 raft_consensus.cc:729] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:25.321449  7505 consensus_queue.cc:260] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.321734  7507 tablet_bootstrap.cc:654] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:25.322113  7505 ts_tablet_manager.cc:1398] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:11:25.322536  7505 tablet_bootstrap.cc:492] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:25.324609  7505 tablet_bootstrap.cc:654] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:25.330070  7504 tablet_bootstrap.cc:492] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:25.330114  7507 tablet_bootstrap.cc:492] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:25.330173  7505 tablet_bootstrap.cc:492] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:25.330250  7507 ts_tablet_manager.cc:1367] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.003s
I20240501 08:11:25.330250  7504 ts_tablet_manager.cc:1367] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.007s
I20240501 08:11:25.330339  7505 ts_tablet_manager.cc:1367] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.004s
I20240501 08:11:25.330952  7505 raft_consensus.cc:348] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.331142  7505 raft_consensus.cc:374] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:25.331015  7507 raft_consensus.cc:348] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.331019  7504 raft_consensus.cc:348] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.331238  7505 raft_consensus.cc:729] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:25.331247  7507 raft_consensus.cc:374] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:25.331260  7504 raft_consensus.cc:374] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:25.331343  7504 raft_consensus.cc:729] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:25.331343  7507 raft_consensus.cc:729] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:25.331509  7505 consensus_queue.cc:260] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.331565  7507 consensus_queue.cc:260] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.331588  7504 consensus_queue.cc:260] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.331990  7505 ts_tablet_manager.cc:1398] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:11:25.332216  7507 ts_tablet_manager.cc:1398] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240501 08:11:25.332326  7505 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:25.333375  7504 ts_tablet_manager.cc:1398] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.002s
I20240501 08:11:25.333628  7504 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:25.333878  7505 tablet_bootstrap.cc:654] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:25.334898  7504 tablet_bootstrap.cc:654] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:25.339362  7504 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:25.339367  7505 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:25.339497  7505 ts_tablet_manager.cc:1367] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.004s
I20240501 08:11:25.339515  7504 ts_tablet_manager.cc:1367] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.004s
I20240501 08:11:25.340227  7505 raft_consensus.cc:348] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:25.340206  7504 raft_consensus.cc:348] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:25.340420  7504 raft_consensus.cc:374] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:25.340402  7505 raft_consensus.cc:374] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:25.340495  7504 raft_consensus.cc:729] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:25.340512  7505 raft_consensus.cc:729] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:25.340718  7505 consensus_queue.cc:260] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:25.340730  7504 consensus_queue.cc:260] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:25.341167  7504 ts_tablet_manager.cc:1398] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:11:25.341202  7505 ts_tablet_manager.cc:1398] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:11:25.427157  7485 raft_consensus.cc:482] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:25.427398  7485 raft_consensus.cc:504] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.428092  7485 leader_election.cc:288] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:25.434666  6542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc900308d554b08b5e54974c0ae77e9" candidate_uuid: "7fb652773c524f639dba8c189b7743ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0190f14862148f4bde78afbbf1ba43d" is_pre_election: true
I20240501 08:11:25.435117  6542 raft_consensus.cc:2454] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb652773c524f639dba8c189b7743ba in term 0.
I20240501 08:11:25.435649  6735 leader_election.cc:302] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:25.435992  7485 raft_consensus.cc:2781] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:25.436089  7485 raft_consensus.cc:482] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:25.436228  7485 raft_consensus.cc:3037] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:25.438077  7485 raft_consensus.cc:504] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.438561  7485 leader_election.cc:288] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 election: Requested vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:25.438987  6542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc900308d554b08b5e54974c0ae77e9" candidate_uuid: "7fb652773c524f639dba8c189b7743ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
I20240501 08:11:25.439148  6542 raft_consensus.cc:3037] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:25.441349  6542 raft_consensus.cc:2454] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb652773c524f639dba8c189b7743ba in term 1.
I20240501 08:11:25.441717  6735 leader_election.cc:302] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:25.441996  7485 raft_consensus.cc:2781] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:25.442174  7485 raft_consensus.cc:686] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Becoming Leader. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Running, Role: LEADER
I20240501 08:11:25.442422  7485 consensus_queue.cc:237] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.445077  6400 catalog_manager.cc:5484] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb652773c524f639dba8c189b7743ba (127.2.248.131). New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:25.499066  7413 raft_consensus.cc:482] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:25.499249  7413 raft_consensus.cc:504] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.499814  7413 leader_election.cc:288] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:25.500485  6672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366e736f8763465299980fe09c0159bf" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654" is_pre_election: true
I20240501 08:11:25.500725  6672 raft_consensus.cc:2454] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:25.501209  6476 leader_election.cc:302] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:25.501485  7413 raft_consensus.cc:2781] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:25.501610  7413 raft_consensus.cc:482] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:25.501708  7413 raft_consensus.cc:3037] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:25.503857  7413 raft_consensus.cc:504] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.504382  7413 leader_election.cc:288] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:25.504943  6672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366e736f8763465299980fe09c0159bf" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:25.505160  6672 raft_consensus.cc:3037] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:25.507055  6672 raft_consensus.cc:2454] T 366e736f8763465299980fe09c0159bf P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:25.507440  6476 leader_election.cc:302] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:25.507740  7413 raft_consensus.cc:2781] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:25.507887  7413 raft_consensus.cc:686] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:25.508193  7413 consensus_queue.cc:237] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:25.510618  6400 catalog_manager.cc:5484] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } }
W20240501 08:11:25.569993  6478 consensus_peers.cc:485] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:11:25.675845  7413 raft_consensus.cc:482] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:25.676069  7413 raft_consensus.cc:504] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:25.676810  7413 leader_election.cc:288] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:25.681957  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff918bc67204080bf1c272c2d950045" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba" is_pre_election: true
I20240501 08:11:25.682163  6802 raft_consensus.cc:2454] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:25.682560  6476 leader_election.cc:302] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:25.682868  7413 raft_consensus.cc:2781] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:25.682955  7413 raft_consensus.cc:482] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:25.683028  7413 raft_consensus.cc:3037] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:25.685169  7413 raft_consensus.cc:504] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:25.685623  7413 leader_election.cc:288] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:25.686084  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff918bc67204080bf1c272c2d950045" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba"
I20240501 08:11:25.686272  6802 raft_consensus.cc:3037] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:25.688055  6802 raft_consensus.cc:2454] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:25.688473  6476 leader_election.cc:302] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:25.688740  7413 raft_consensus.cc:2781] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:25.688891  7413 raft_consensus.cc:686] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:25.689137  7413 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:25.691519  6400 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:25.944875  7413 consensus_queue.cc:1035] T 366e736f8763465299980fe09c0159bf P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:25.960542  6478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:11:26.065007  7485 consensus_queue.cc:1035] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:26.094843  7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:26.095142  7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:26.102380  7518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:11:26.226027  6802 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:26.226783  7413 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:26.240489  7552 mvcc.cc:205] Tried to move back new op lower bound from 7022801249166835712 to 7022801246984368128. Current Snapshot: MvccSnapshot[applied={T|T < 7022801249166835712}]

INSERT report
    rows total: 2000
    time total: 165.027 ms
  time per row: 0.0825133 ms
I20240501 08:11:26.412364  6398 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34732:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:11:26.420660  6652 tablet_service.cc:1467] Processing CreateTablet for tablet 217060ad409b4492a7d4ce043578aa49 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:26.421154  6652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217060ad409b4492a7d4ce043578aa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:26.421342  6782 tablet_service.cc:1467] Processing CreateTablet for tablet 89727cd975064b5680ff992414dbaef0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:26.421744  6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89727cd975064b5680ff992414dbaef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:26.424574  6781 tablet_service.cc:1467] Processing CreateTablet for tablet 217060ad409b4492a7d4ce043578aa49 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:26.424914  6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217060ad409b4492a7d4ce043578aa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:26.427426  6520 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc205621c874d5ab791bb3329988ccc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:26.427857  6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc205621c874d5ab791bb3329988ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:26.428445  7571 tablet_bootstrap.cc:492] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:26.431290  7571 tablet_bootstrap.cc:654] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:26.431291  6651 tablet_service.cc:1467] Processing CreateTablet for tablet 89727cd975064b5680ff992414dbaef0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:26.431684  6651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89727cd975064b5680ff992414dbaef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:26.434639  7570 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:26.437090  7573 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:26.437599  6650 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc205621c874d5ab791bb3329988ccc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:26.437944  6650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc205621c874d5ab791bb3329988ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:26.438277  7570 tablet_bootstrap.cc:654] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:26.438716  7571 tablet_bootstrap.cc:492] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:26.438881  7571 ts_tablet_manager.cc:1367] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.006s
I20240501 08:11:26.438947  7573 tablet_bootstrap.cc:654] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:26.439657  7571 raft_consensus.cc:348] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.439935  7571 raft_consensus.cc:374] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:26.440070  7571 raft_consensus.cc:729] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:26.440399  7571 consensus_queue.cc:260] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.441130  7571 ts_tablet_manager.cc:1398] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:26.441762  7571 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:26.443320  7571 tablet_bootstrap.cc:654] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:26.454838  7570 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:26.455053  7570 ts_tablet_manager.cc:1367] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.011s
I20240501 08:11:26.455253  7571 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:26.455417  7571 ts_tablet_manager.cc:1367] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.001s
I20240501 08:11:26.455756  7573 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:26.455919  7573 ts_tablet_manager.cc:1367] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.019s	user 0.002s	sys 0.016s
I20240501 08:11:26.455958  7570 raft_consensus.cc:348] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:26.456184  7570 raft_consensus.cc:374] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:26.456248  7571 raft_consensus.cc:348] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:26.456336  7570 raft_consensus.cc:729] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:26.456455  7571 raft_consensus.cc:374] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:26.456532  7571 raft_consensus.cc:729] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:26.456610  7570 consensus_queue.cc:260] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:26.456712  7573 raft_consensus.cc:348] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.456756  7571 consensus_queue.cc:260] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:26.457235  7571 ts_tablet_manager.cc:1398] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:11:26.457401  7573 raft_consensus.cc:374] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:26.457515  7573 raft_consensus.cc:729] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:26.457734  7573 consensus_queue.cc:260] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.457891  7570 ts_tablet_manager.cc:1398] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:11:26.458328  7570 tablet_bootstrap.cc:492] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:26.458343  7573 ts_tablet_manager.cc:1398] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:26.459973  7570 tablet_bootstrap.cc:654] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:26.463176  7570 tablet_bootstrap.cc:492] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:26.463356  7570 ts_tablet_manager.cc:1367] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240501 08:11:26.463927  7570 raft_consensus.cc:348] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.464068  7570 raft_consensus.cc:374] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:26.464154  7570 raft_consensus.cc:729] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:26.464332  7570 consensus_queue.cc:260] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.464715  7570 ts_tablet_manager.cc:1398] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240501 08:11:26.464999  7570 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:26.466455  7570 tablet_bootstrap.cc:654] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:26.468283  7570 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:26.468406  7570 ts_tablet_manager.cc:1367] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:11:26.468878  7570 raft_consensus.cc:348] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.469007  7570 raft_consensus.cc:374] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:26.469092  7570 raft_consensus.cc:729] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:26.469540  7570 consensus_queue.cc:260] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.470064  7570 ts_tablet_manager.cc:1398] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:26.539326  7430 raft_consensus.cc:482] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:26.539498  7430 raft_consensus.cc:504] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.539983  7430 leader_election.cc:288] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:26.540666  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89727cd975064b5680ff992414dbaef0" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba" is_pre_election: true
I20240501 08:11:26.540910  6802 raft_consensus.cc:2454] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 0.
I20240501 08:11:26.541304  6606 leader_election.cc:302] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:26.541584  7430 raft_consensus.cc:2781] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:26.541692  7430 raft_consensus.cc:482] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:26.541780  7430 raft_consensus.cc:3037] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:26.543697  7430 raft_consensus.cc:504] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.544096  7430 leader_election.cc:288] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Requested vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:26.544606  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89727cd975064b5680ff992414dbaef0" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba"
I20240501 08:11:26.544806  6802 raft_consensus.cc:3037] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:26.546572  6802 raft_consensus.cc:2454] T 89727cd975064b5680ff992414dbaef0 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 1.
I20240501 08:11:26.546924  6606 leader_election.cc:302] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:26.547200  7430 raft_consensus.cc:2781] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:26.547590  7430 raft_consensus.cc:686] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Becoming Leader. State: Replica: d143a9ab17334a6081de63802bdce654, State: Running, Role: LEADER
I20240501 08:11:26.547857  7430 consensus_queue.cc:237] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.550159  6398 catalog_manager.cc:5484] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 reported cstate change: term changed from 0 to 1, leader changed from <none> to d143a9ab17334a6081de63802bdce654 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:26.575277  7435 raft_consensus.cc:482] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:26.575444  7435 raft_consensus.cc:504] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.576022  7435 leader_election.cc:288] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:26.576642  6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654" is_pre_election: true
I20240501 08:11:26.576857  6671 raft_consensus.cc:2454] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:26.577363  6476 leader_election.cc:302] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:26.577646  7435 raft_consensus.cc:2781] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:26.577754  7435 raft_consensus.cc:482] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:26.577817  7435 raft_consensus.cc:3037] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:26.579597  7435 raft_consensus.cc:504] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.580003  7435 leader_election.cc:288] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:26.580492  6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:26.580683  6671 raft_consensus.cc:3037] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:26.582703  6671 raft_consensus.cc:2454] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:26.583109  6476 leader_election.cc:302] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:26.583392  7435 raft_consensus.cc:2781] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:26.583514  7435 raft_consensus.cc:686] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:26.583767  7435 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:26.586428  6397 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:26.691948  7430 raft_consensus.cc:482] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:26.692199  7430 raft_consensus.cc:504] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:26.692695  7430 leader_election.cc:288] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:26.693437  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217060ad409b4492a7d4ce043578aa49" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba" is_pre_election: true
I20240501 08:11:26.693709  6802 raft_consensus.cc:2454] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 0.
I20240501 08:11:26.694135  6606 leader_election.cc:302] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:26.694461  7430 raft_consensus.cc:2781] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:26.694553  7430 raft_consensus.cc:482] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:26.694653  7430 raft_consensus.cc:3037] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:26.696441  7430 raft_consensus.cc:504] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:26.696830  7430 leader_election.cc:288] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Requested vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:26.697373  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217060ad409b4492a7d4ce043578aa49" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba"
I20240501 08:11:26.697535  6802 raft_consensus.cc:3037] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:26.699220  6802 raft_consensus.cc:2454] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 1.
I20240501 08:11:26.699590  6606 leader_election.cc:302] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:26.699863  7430 raft_consensus.cc:2781] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:26.700023  7430 raft_consensus.cc:686] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Becoming Leader. State: Replica: d143a9ab17334a6081de63802bdce654, State: Running, Role: LEADER
I20240501 08:11:26.700284  7430 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:26.702461  6397 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 reported cstate change: term changed from 0 to 1, leader changed from <none> to d143a9ab17334a6081de63802bdce654 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:26.989411  7430 consensus_queue.cc:1035] T 89727cd975064b5680ff992414dbaef0 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:27.053026  7583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:27.053400  7583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:27.061578  7583 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 08:11:27.147869  7430 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:27.172854  7435 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 168.057 ms
  time per row: 0.0840284 ms
W20240501 08:11:27.871157  6478 consensus_peers.cc:485] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:11:28.293761  6400 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34732:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:11:28.302721  6781 tablet_service.cc:1467] Processing CreateTablet for tablet 527072daf1ea4b478a1b35f4a5c4bf3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:28.302966  6520 tablet_service.cc:1467] Processing CreateTablet for tablet 527072daf1ea4b478a1b35f4a5c4bf3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:28.303220  6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527072daf1ea4b478a1b35f4a5c4bf3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:28.303401  6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527072daf1ea4b478a1b35f4a5c4bf3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:28.303320  6782 tablet_service.cc:1467] Processing CreateTablet for tablet f2b0f6861d824903ab421529fdd2e55a (DEFAULT_TABLE table=rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:28.303694  6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b0f6861d824903ab421529fdd2e55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:28.304212  6521 tablet_service.cc:1467] Processing CreateTablet for tablet f2b0f6861d824903ab421529fdd2e55a (DEFAULT_TABLE table=rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:28.304562  6521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b0f6861d824903ab421529fdd2e55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:28.308571  6650 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbcd048e5a94589a1ea17c449688af4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:28.308969  6650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbcd048e5a94589a1ea17c449688af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:28.309001  6522 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbcd048e5a94589a1ea17c449688af4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:28.309348  6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbcd048e5a94589a1ea17c449688af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:28.319238  7640 tablet_bootstrap.cc:492] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:28.321135  7639 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:28.321695  7640 tablet_bootstrap.cc:654] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:28.323446  7639 tablet_bootstrap.cc:654] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:28.323500  7640 tablet_bootstrap.cc:492] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:28.323632  7640 ts_tablet_manager.cc:1367] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 08:11:28.324379  7640 raft_consensus.cc:348] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.324577  7640 raft_consensus.cc:374] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:28.324669  7640 raft_consensus.cc:729] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:28.324972  7640 consensus_queue.cc:260] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.325877  7640 ts_tablet_manager.cc:1398] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:11:28.326172  7640 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:28.327548  7640 tablet_bootstrap.cc:654] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:28.342720  7642 tablet_bootstrap.cc:492] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:28.345302  7642 tablet_bootstrap.cc:654] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:28.345405  7640 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:28.345537  7639 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:28.345618  7640 ts_tablet_manager.cc:1367] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.019s	user 0.003s	sys 0.013s
I20240501 08:11:28.345710  7639 ts_tablet_manager.cc:1367] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.008s
I20240501 08:11:28.346364  7640 raft_consensus.cc:348] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.346382  7639 raft_consensus.cc:348] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.346589  7639 raft_consensus.cc:374] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:28.346609  7640 raft_consensus.cc:374] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:28.346684  7640 raft_consensus.cc:729] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:28.346693  7639 raft_consensus.cc:729] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:28.346931  7639 consensus_queue.cc:260] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.346967  7640 consensus_queue.cc:260] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.347102  7642 tablet_bootstrap.cc:492] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:28.347251  7642 ts_tablet_manager.cc:1367] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 08:11:28.347565  7640 ts_tablet_manager.cc:1398] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:11:28.347576  7639 ts_tablet_manager.cc:1398] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:11:28.347813  7639 tablet_bootstrap.cc:492] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:28.348181  7642 raft_consensus.cc:348] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.348376  7642 raft_consensus.cc:374] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:28.348470  7642 raft_consensus.cc:729] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:28.349634  7639 tablet_bootstrap.cc:654] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:28.348706  7642 consensus_queue.cc:260] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.350471  7642 ts_tablet_manager.cc:1398] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:11:28.352519  7639 tablet_bootstrap.cc:492] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:28.352669  7639 ts_tablet_manager.cc:1367] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 08:11:28.353283  7639 raft_consensus.cc:348] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.353446  7639 raft_consensus.cc:374] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:28.353525  7639 raft_consensus.cc:729] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:28.353749  7639 consensus_queue.cc:260] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.354235  7639 ts_tablet_manager.cc:1398] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:11:28.354599  7639 tablet_bootstrap.cc:492] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:28.356209  7639 tablet_bootstrap.cc:654] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:28.357959  7639 tablet_bootstrap.cc:492] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:28.358099  7639 ts_tablet_manager.cc:1367] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:11:28.358642  7639 raft_consensus.cc:348] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.358789  7639 raft_consensus.cc:374] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:28.358860  7639 raft_consensus.cc:729] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:28.359037  7639 consensus_queue.cc:260] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.359436  7639 ts_tablet_manager.cc:1398] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:11:28.561266  7435 raft_consensus.cc:482] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:28.561452  7435 raft_consensus.cc:504] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.561934  7435 leader_election.cc:288] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:28.562565  6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b0f6861d824903ab421529fdd2e55a" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba" is_pre_election: true
I20240501 08:11:28.562758  6801 raft_consensus.cc:2454] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:28.563189  6476 leader_election.cc:302] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:28.563439  7435 raft_consensus.cc:2781] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:28.563542  7435 raft_consensus.cc:482] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:28.563606  7435 raft_consensus.cc:3037] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:28.565433  7435 raft_consensus.cc:504] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.565824  7435 leader_election.cc:288] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:28.566287  6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b0f6861d824903ab421529fdd2e55a" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba"
I20240501 08:11:28.566458  6801 raft_consensus.cc:3037] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:28.568460  6801 raft_consensus.cc:2454] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:28.568825  6476 leader_election.cc:302] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:28.569068  7435 raft_consensus.cc:2781] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:28.569396  7435 raft_consensus.cc:686] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:28.569643  7435 consensus_queue.cc:237] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.571971  6400 catalog_manager.cc:5484] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:28.748850  7430 raft_consensus.cc:482] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:28.749032  7430 raft_consensus.cc:504] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.749617  7430 leader_election.cc:288] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:28.755241  6542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbcd048e5a94589a1ea17c449688af4" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0190f14862148f4bde78afbbf1ba43d" is_pre_election: true
I20240501 08:11:28.755501  6542 raft_consensus.cc:2454] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 0.
I20240501 08:11:28.755964  6605 leader_election.cc:302] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:28.756287  7430 raft_consensus.cc:2781] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:28.756390  7430 raft_consensus.cc:482] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:28.756461  7430 raft_consensus.cc:3037] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:28.758297  7430 raft_consensus.cc:504] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.758715  7430 leader_election.cc:288] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Requested vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:28.759166  6542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbcd048e5a94589a1ea17c449688af4" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
I20240501 08:11:28.759346  6542 raft_consensus.cc:3037] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:28.761482  6542 raft_consensus.cc:2454] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 1.
I20240501 08:11:28.761835  6605 leader_election.cc:302] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:28.762125  7430 raft_consensus.cc:2781] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:28.762493  7430 raft_consensus.cc:686] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Becoming Leader. State: Replica: d143a9ab17334a6081de63802bdce654, State: Running, Role: LEADER
I20240501 08:11:28.762766  7430 consensus_queue.cc:237] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.765298  6400 catalog_manager.cc:5484] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 reported cstate change: term changed from 0 to 1, leader changed from <none> to d143a9ab17334a6081de63802bdce654 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:28.777630  7435 raft_consensus.cc:482] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:28.777819  7435 raft_consensus.cc:504] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.778316  7435 leader_election.cc:288] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:28.778824  6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527072daf1ea4b478a1b35f4a5c4bf3c" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba" is_pre_election: true
I20240501 08:11:28.779043  6801 raft_consensus.cc:2454] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:28.779408  6476 leader_election.cc:302] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:28.779665  7435 raft_consensus.cc:2781] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:28.779749  7435 raft_consensus.cc:482] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:28.779841  7435 raft_consensus.cc:3037] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:28.781780  7435 raft_consensus.cc:504] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.782202  7435 leader_election.cc:288] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:28.782635  6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527072daf1ea4b478a1b35f4a5c4bf3c" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba"
I20240501 08:11:28.782827  6801 raft_consensus.cc:3037] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:28.784788  6801 raft_consensus.cc:2454] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:28.785181  6476 leader_election.cc:302] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:28.785442  7435 raft_consensus.cc:2781] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:28.785619  7435 raft_consensus.cc:686] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:28.785893  7435 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } }
I20240501 08:11:28.788650  6400 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:29.060511  7435 consensus_queue.cc:1035] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:29.173507  7430 consensus_queue.cc:1035] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:29.212492  7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:29.212867  7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:29.220659  7657 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 08:11:29.349442  6801 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:29.350059  7435 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:29.401664  7660 mvcc.cc:205] Tried to move back new op lower bound from 7022801261967851520 to 7022801259668537344. Current Snapshot: MvccSnapshot[applied={T|T < 7022801261967851520}]

INSERT report
    rows total: 2000
    time total: 177.376 ms
  time per row: 0.0886881 ms
W20240501 08:11:30.342743  6478 consensus_peers.cc:485] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:11:30.461779  6396 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34732:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:11:30.470650  6650 tablet_service.cc:1467] Processing CreateTablet for tablet 97da0f9e56f14f6a985b6ffe8c44b68c (DEFAULT_TABLE table=rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:30.471196  6650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97da0f9e56f14f6a985b6ffe8c44b68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.471311  6651 tablet_service.cc:1467] Processing CreateTablet for tablet 74cea6ae2bd446ce8adc7442dea7f030 (DEFAULT_TABLE table=rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:30.471439  6782 tablet_service.cc:1467] Processing CreateTablet for tablet 97da0f9e56f14f6a985b6ffe8c44b68c (DEFAULT_TABLE table=rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:30.471707  6651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cea6ae2bd446ce8adc7442dea7f030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.471680  6522 tablet_service.cc:1467] Processing CreateTablet for tablet 27f9665553394cf4a53e8ebc82fbc575 (DEFAULT_TABLE table=rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:30.471827  6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97da0f9e56f14f6a985b6ffe8c44b68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.472067  6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f9665553394cf4a53e8ebc82fbc575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.477571  6652 tablet_service.cc:1467] Processing CreateTablet for tablet 27f9665553394cf4a53e8ebc82fbc575 (DEFAULT_TABLE table=rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:30.478009  6652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f9665553394cf4a53e8ebc82fbc575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.479080  7721 tablet_bootstrap.cc:492] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:30.478843  7720 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:30.480196  6781 tablet_service.cc:1467] Processing CreateTablet for tablet 74cea6ae2bd446ce8adc7442dea7f030 (DEFAULT_TABLE table=rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:30.480571  6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cea6ae2bd446ce8adc7442dea7f030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.480835  7721 tablet_bootstrap.cc:654] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:30.480904  7720 tablet_bootstrap.cc:654] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:30.486758  7720 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:30.486779  7721 tablet_bootstrap.cc:492] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:30.486912  7721 ts_tablet_manager.cc:1367] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240501 08:11:30.486935  7720 ts_tablet_manager.cc:1367] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.008s	user 0.008s	sys 0.000s
I20240501 08:11:30.487656  7720 raft_consensus.cc:348] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.487744  7721 raft_consensus.cc:348] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.487957  7720 raft_consensus.cc:374] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:30.488051  7721 raft_consensus.cc:374] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:30.488173  7720 raft_consensus.cc:729] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:30.488303  7721 raft_consensus.cc:729] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:30.488483  7720 consensus_queue.cc:260] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.488646  7721 consensus_queue.cc:260] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.489205  7720 ts_tablet_manager.cc:1398] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:30.489382  7721 ts_tablet_manager.cc:1398] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:11:30.489549  7720 tablet_bootstrap.cc:492] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:30.490336  7724 tablet_bootstrap.cc:492] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d: Bootstrap starting.
I20240501 08:11:30.491153  7720 tablet_bootstrap.cc:654] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:30.491971  7724 tablet_bootstrap.cc:654] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:30.493821  7721 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: Bootstrap starting.
I20240501 08:11:30.497566  7721 tablet_bootstrap.cc:654] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:30.499948  7724 tablet_bootstrap.cc:492] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d: No bootstrap required, opened a new log
I20240501 08:11:30.500130  7724 ts_tablet_manager.cc:1367] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240501 08:11:30.500321  7721 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: No bootstrap required, opened a new log
I20240501 08:11:30.500475  7721 ts_tablet_manager.cc:1367] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.004s
I20240501 08:11:30.500586  7720 tablet_bootstrap.cc:492] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:30.500758  7720 ts_tablet_manager.cc:1367] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.004s
I20240501 08:11:30.500890  7724 raft_consensus.cc:348] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:30.501161  7724 raft_consensus.cc:374] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:30.501103  7721 raft_consensus.cc:348] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.501293  7724 raft_consensus.cc:729] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Initialized, Role: FOLLOWER
I20240501 08:11:30.501340  7721 raft_consensus.cc:374] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:30.501430  7721 raft_consensus.cc:729] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Initialized, Role: FOLLOWER
I20240501 08:11:30.501529  7720 raft_consensus.cc:348] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.501578  7724 consensus_queue.cc:260] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:30.501742  7720 raft_consensus.cc:374] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:30.501667  7721 consensus_queue.cc:260] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.501824  7720 raft_consensus.cc:729] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:30.502091  7720 consensus_queue.cc:260] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.502188  7721 ts_tablet_manager.cc:1398] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:11:30.502630  7720 ts_tablet_manager.cc:1398] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:30.502982  7724 ts_tablet_manager.cc:1398] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:11:30.503753  7720 tablet_bootstrap.cc:492] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: Bootstrap starting.
I20240501 08:11:30.505497  7720 tablet_bootstrap.cc:654] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:30.507639  7720 tablet_bootstrap.cc:492] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: No bootstrap required, opened a new log
I20240501 08:11:30.507828  7720 ts_tablet_manager.cc:1367] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:11:30.508495  7720 raft_consensus.cc:348] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:30.508641  7720 raft_consensus.cc:374] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:30.508744  7720 raft_consensus.cc:729] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Initialized, Role: FOLLOWER
I20240501 08:11:30.508976  7720 consensus_queue.cc:260] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:30.509487  7720 ts_tablet_manager.cc:1398] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:11:30.578408  7435 raft_consensus.cc:482] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:30.578609  7435 raft_consensus.cc:504] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:30.579144  7435 leader_election.cc:288] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:30.579762  6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f9665553394cf4a53e8ebc82fbc575" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654" is_pre_election: true
I20240501 08:11:30.579998  6671 raft_consensus.cc:2454] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 0.
I20240501 08:11:30.580484  6476 leader_election.cc:302] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:30.580762  7435 raft_consensus.cc:2781] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:30.580857  7435 raft_consensus.cc:482] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:30.580940  7435 raft_consensus.cc:3037] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:30.582810  7435 raft_consensus.cc:504] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:30.583271  7435 leader_election.cc:288] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:30.583854  6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f9665553394cf4a53e8ebc82fbc575" candidate_uuid: "b0190f14862148f4bde78afbbf1ba43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:30.584018  6671 raft_consensus.cc:3037] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:30.585951  6671 raft_consensus.cc:2454] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0190f14862148f4bde78afbbf1ba43d in term 1.
I20240501 08:11:30.586324  6476 leader_election.cc:302] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:30.586587  7435 raft_consensus.cc:2781] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:30.586918  7435 raft_consensus.cc:686] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Leader. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:30.587370  7435 consensus_queue.cc:237] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } }
I20240501 08:11:30.589906  6401 catalog_manager.cc:5484] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0190f14862148f4bde78afbbf1ba43d (127.2.248.129). New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:30.626400  7641 raft_consensus.cc:482] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:30.626575  7641 raft_consensus.cc:504] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.627189  7641 leader_election.cc:288] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:30.627818  6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cea6ae2bd446ce8adc7442dea7f030" candidate_uuid: "7fb652773c524f639dba8c189b7743ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654" is_pre_election: true
I20240501 08:11:30.628033  6671 raft_consensus.cc:2454] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb652773c524f639dba8c189b7743ba in term 0.
I20240501 08:11:30.628537  6736 leader_election.cc:302] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:30.628819  7641 raft_consensus.cc:2781] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:30.628937  7641 raft_consensus.cc:482] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:30.628995  7641 raft_consensus.cc:3037] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:30.630867  7641 raft_consensus.cc:504] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.631326  7641 leader_election.cc:288] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:30.631893  6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cea6ae2bd446ce8adc7442dea7f030" candidate_uuid: "7fb652773c524f639dba8c189b7743ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:30.632160  6671 raft_consensus.cc:3037] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:30.634055  6671 raft_consensus.cc:2454] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb652773c524f639dba8c189b7743ba in term 1.
I20240501 08:11:30.634490  6736 leader_election.cc:302] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:30.634810  7641 raft_consensus.cc:2781] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:30.635250  7641 raft_consensus.cc:686] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Becoming Leader. State: Replica: 7fb652773c524f639dba8c189b7743ba, State: Running, Role: LEADER
I20240501 08:11:30.635591  7641 consensus_queue.cc:237] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.638295  6401 catalog_manager.cc:5484] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb652773c524f639dba8c189b7743ba (127.2.248.131). New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:30.664949  7430 raft_consensus.cc:482] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:30.665161  7430 raft_consensus.cc:504] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.665729  7430 leader_election.cc:288] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:30.666291  6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97da0f9e56f14f6a985b6ffe8c44b68c" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba" is_pre_election: true
I20240501 08:11:30.666536  6801 raft_consensus.cc:2454] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 0.
I20240501 08:11:30.666998  6606 leader_election.cc:302] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:30.667273  7430 raft_consensus.cc:2781] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:30.667387  7430 raft_consensus.cc:482] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:30.667465  7430 raft_consensus.cc:3037] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:30.670284  7430 raft_consensus.cc:504] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.670818  7430 leader_election.cc:288] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Requested vote from peers 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:30.671403  6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97da0f9e56f14f6a985b6ffe8c44b68c" candidate_uuid: "d143a9ab17334a6081de63802bdce654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb652773c524f639dba8c189b7743ba"
I20240501 08:11:30.671628  6801 raft_consensus.cc:3037] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:30.673663  6801 raft_consensus.cc:2454] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d143a9ab17334a6081de63802bdce654 in term 1.
I20240501 08:11:30.674062  6606 leader_election.cc:302] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7fb652773c524f639dba8c189b7743ba, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:30.674386  7430 raft_consensus.cc:2781] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:30.674726  7430 raft_consensus.cc:686] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Becoming Leader. State: Replica: d143a9ab17334a6081de63802bdce654, State: Running, Role: LEADER
I20240501 08:11:30.675015  7430 consensus_queue.cc:237] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } }
I20240501 08:11:30.677518  6401 catalog_manager.cc:5484] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 reported cstate change: term changed from 0 to 1, leader changed from <none> to d143a9ab17334a6081de63802bdce654 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:31.004143  7735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:31.004416  7735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:31.011631  7735 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:11:31.092725  7435 consensus_queue.cc:1035] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:31.186228  6672 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:31.187299  7641 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:31.196787  6801 raft_consensus.cc:1261] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:31.197481  7733 consensus_queue.cc:1035] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:31.224671  7775 mvcc.cc:205] Tried to move back new op lower bound from 7022801269491511296 to 7022801267244871680. Current Snapshot: MvccSnapshot[applied={T|T < 7022801269491511296}]

INSERT report
    rows total: 2000
    time total: 203.571 ms
  time per row: 0.101785 ms
W20240501 08:11:31.353883  6478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111) [suppressed 10 similar messages]
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:11:32.280885  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:36129
--local_ip_for_outbound_sockets=127.2.248.132
--tserver_master_addrs=127.2.248.190:39497
--webserver_port=43137
--webserver_interface=127.2.248.132
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:11:32.608093  7791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:32.608413  7791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:32.608544  7791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:32.616155  7791 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:32.616287  7791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:32.616443  7791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:11:32.625030  7791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:36129
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=43137
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:32.626348  7791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:32.628024  7791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:32.639982  7797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:32.640574  7796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:32.641052  7799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:32.641795  7791 server_base.cc:1034] running on GCE node
I20240501 08:11:32.642354  7791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:32.643277  7791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:32.644471  7791 hybrid_clock.cc:648] HybridClock initialized: now 1714551092644410 us; error 50 us; skew 500 ppm
I20240501 08:11:32.644775  7791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:32.647197  7791 webserver.cc:460] Webserver started at http://127.2.248.132:43137/ using document root <none> and password file <none>
I20240501 08:11:32.648394  7791 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:32.648561  7791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20240501 08:11:32.648514  6478 consensus_peers.cc:485] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:11:32.654134  7791 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.004s
I20240501 08:11:32.656791  7806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:32.658195  7791 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:11:32.658354  7791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fcfccbafc3254a87896f0e72c8078ab5"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:32.659237  7791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:32.678272  7791 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:32.679003  7791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:32.679270  7791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:32.680138  7791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:32.681742  7791 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:32.681813  7791 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:32.681921  7791 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:32.681967  7791 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:32.710330  7791 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:36129
I20240501 08:11:32.710376  7918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:36129 every 8 connection(s)
I20240501 08:11:32.711691  7791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:32.712674  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 7791
I20240501 08:11:32.713408  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:43177
--local_ip_for_outbound_sockets=127.2.248.133
--tserver_master_addrs=127.2.248.190:39497
--webserver_port=40057
--webserver_interface=127.2.248.133
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:32.726570  7919 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:32.726919  7919 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:32.727900  7919 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:32.729665  6394 ts_manager.cc:194] Re-registered known tserver with Master: fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129)
I20240501 08:11:32.730547  6394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:60029
W20240501 08:11:33.036729  7923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:33.037003  7923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:33.037142  7923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:33.044852  7923 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:33.044981  7923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:33.045154  7923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:11:33.054327  7923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:43177
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=40057
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:33.055670  7923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:33.057504  7923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:33.069447  7928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.070120  7929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.070669  7931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:33.071457  7923 server_base.cc:1034] running on GCE node
I20240501 08:11:33.072021  7923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:33.072943  7923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:33.074157  7923 hybrid_clock.cc:648] HybridClock initialized: now 1714551093074094 us; error 63 us; skew 500 ppm
I20240501 08:11:33.074560  7923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:33.076897  7923 webserver.cc:460] Webserver started at http://127.2.248.133:40057/ using document root <none> and password file <none>
I20240501 08:11:33.077605  7923 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:33.077705  7923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:33.082785  7923 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240501 08:11:33.085628  7937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:33.086850  7923 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:11:33.086983  7923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0257c0e8038d4e3fa9580cb47278f16f"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-0pdg"
I20240501 08:11:33.087821  7923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:33.106940  7923 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:33.107758  7923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:33.108002  7923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:33.108947  7923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:33.110735  7944 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 08:11:33.116078  7923 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 08:11:33.116202  7923 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20240501 08:11:33.116308  7923 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 08:11:33.119992  7923 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 08:11:33.120150  7923 ts_tablet_manager.cc:569] Time spent register tablets: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:11:33.120537  7944 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap starting.
I20240501 08:11:33.143412  7944 log.cc:824] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:33.147035  7944 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:33.147640  7944 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap complete.
I20240501 08:11:33.149058  7944 ts_tablet_manager.cc:1367] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Time spent bootstrapping tablet: real 0.029s	user 0.021s	sys 0.007s
I20240501 08:11:33.159873  7944 raft_consensus.cc:348] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:33.161991  7944 raft_consensus.cc:729] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0257c0e8038d4e3fa9580cb47278f16f, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.163247  7944 consensus_queue.cc:260] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } }
I20240501 08:11:33.169718  7944 ts_tablet_manager.cc:1398] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.011s
I20240501 08:11:33.183432  7923 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:43177
I20240501 08:11:33.183516  8052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:43177 every 8 connection(s)
I20240501 08:11:33.184944  7923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
W20240501 08:11:33.186138  8054 tablet.cc:2369] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:33.189182  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 7923
I20240501 08:11:33.189922  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.134:33609
--local_ip_for_outbound_sockets=127.2.248.134
--tserver_master_addrs=127.2.248.190:39497
--webserver_port=37369
--webserver_interface=127.2.248.134
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:33.197109  8053 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:33.197435  8053 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:33.198393  8053 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:33.201051  6394 ts_manager.cc:194] Re-registered known tserver with Master: 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177)
I20240501 08:11:33.202139  6394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:34355
W20240501 08:11:33.512712  8056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:33.512956  8056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:33.513067  8056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:33.519974  8056 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:33.520079  8056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:33.520251  8056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.134
I20240501 08:11:33.528936  8056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.134:33609
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.248.134
--webserver_port=37369
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.134
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:33.530139  8056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:33.531773  8056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:33.542922  8064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.543571  8061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.544236  8062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:33.544939  8056 server_base.cc:1034] running on GCE node
I20240501 08:11:33.545550  8056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:33.546618  8056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:33.547813  8056 hybrid_clock.cc:648] HybridClock initialized: now 1714551093547764 us; error 44 us; skew 500 ppm
I20240501 08:11:33.548079  8056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:33.550150  8056 webserver.cc:460] Webserver started at http://127.2.248.134:37369/ using document root <none> and password file <none>
I20240501 08:11:33.550801  8056 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:33.550932  8056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:33.555662  8056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:11:33.558122  8070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:33.559337  8056 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:11:33.559473  8056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c284970db26b4c95a9ee09310b5adcbe"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:33.560233  8056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:33.577422  8056 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:33.578099  8056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:33.578370  8056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:33.579201  8056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:33.580602  8056 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:33.580698  8056 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:33.580817  8056 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:33.580869  8056 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:33.614987  8056 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.134:33609
I20240501 08:11:33.615028  8182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.134:33609 every 8 connection(s)
I20240501 08:11:33.616379  8056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:33.623111  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 8056
I20240501 08:11:33.623698  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.135:36997
--local_ip_for_outbound_sockets=127.2.248.135
--tserver_master_addrs=127.2.248.190:39497
--webserver_port=40711
--webserver_interface=127.2.248.135
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:33.628778  8183 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:33.629125  8183 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:33.630076  8183 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:33.631608  6394 ts_manager.cc:194] Re-registered known tserver with Master: c284970db26b4c95a9ee09310b5adcbe (127.2.248.134:33609)
I20240501 08:11:33.632731  6394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.134:35867
I20240501 08:11:33.689728  8053 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
I20240501 08:11:33.733595  7919 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
W20240501 08:11:33.941784  8186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:33.942015  8186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:33.942106  8186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:33.948963  8186 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:33.949064  8186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:33.949178  8186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.135
I20240501 08:11:33.957237  8186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.135:36997
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.248.135
--webserver_port=40711
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.135
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:33.958336  8186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:33.959867  8186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:33.972136  8198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.972637  8199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:33.973368  8201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:33.974149  8186 server_base.cc:1034] running on GCE node
I20240501 08:11:33.974661  8186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:33.975442  8186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:33.976613  8186 hybrid_clock.cc:648] HybridClock initialized: now 1714551093976580 us; error 32 us; skew 500 ppm
I20240501 08:11:33.976861  8186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:33.978837  8186 webserver.cc:460] Webserver started at http://127.2.248.135:40711/ using document root <none> and password file <none>
I20240501 08:11:33.979480  8186 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:33.979598  8186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:33.984643  8186 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:11:33.987035  8207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:33.988562  8186 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:11:33.988714  8186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ad679e52ba5643f2b11838d14696d1ed"
format_stamp: "Formatted at 2024-05-01 08:11:19 on dist-test-slave-0pdg"
I20240501 08:11:33.989533  8186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:34.017562  8186 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:34.018368  8186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:34.018682  8186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:34.019582  8186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:34.021210  8186 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:34.021277  8186 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:34.021397  8186 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:34.021442  8186 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:34.051272  8186 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.135:36997
I20240501 08:11:34.051278  8319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.135:36997 every 8 connection(s)
I20240501 08:11:34.053027  8186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:34.057713  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 8186
I20240501 08:11:34.058005  3042 test_util.cc:268] Using random seed: 149494762
I20240501 08:11:34.067781  8320 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:34.068051  8320 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:34.068773  8320 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:34.070204  6394 ts_manager.cc:194] Re-registered known tserver with Master: ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997)
I20240501 08:11:34.071161  6394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.135:33485
I20240501 08:11:34.076232  3042 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 08:11:34.076664  3042 test_util.cc:268] Using random seed: 149513432
I20240501 08:11:34.090921  3042 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 08:11:34.091759  3042 test_util.cc:268] Using random seed: 149528491
I20240501 08:11:34.126118  3042 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 08:11:34.126515  3042 test_util.cc:268] Using random seed: 149563250
I20240501 08:11:34.229759  3042 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 08:11:34.231752  3042 test_util.cc:268] Using random seed: 149668457
I20240501 08:11:34.392474  3042 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 08:11:34.635181  8183 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
I20240501 08:11:35.073592  8320 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
I20240501 08:11:35.390823  7986 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:35.426709  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:35.505538  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:35.510322  8118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:35.516561  7854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:35.554085  6522 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:35.567727  6652 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:36.361543  7986 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:36.435300  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:36.437031  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:36.476318  7854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:36.495692  6652 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:36.527499  8118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:36.569654  6522 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:36.868577  6798 consensus_queue.cc:237] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:36.870860  6670 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240501 08:11:36.871696  7641 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240501 08:11:36.875602  7641 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:36.880506  6670 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:36.891822  6398 catalog_manager.cc:5484] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index -1 to 44, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
W20240501 08:11:36.913275  6738 consensus_peers.cc:485] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba -> Peer ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997): Couldn't send request to peer ad679e52ba5643f2b11838d14696d1ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74cea6ae2bd446ce8adc7442dea7f030. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.017906  6671 consensus_queue.cc:237] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:37.024226  6540 raft_consensus.cc:1261] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240501 08:11:37.026419  8487 consensus_queue.cc:1035] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240501 08:11:37.033169  8498 raft_consensus.cc:2932] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:37.034451  6540 raft_consensus.cc:2932] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:37.041100  6401 catalog_manager.cc:5484] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index -1 to 44, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:37.051278  6608 consensus_peers.cc:485] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 -> Peer c284970db26b4c95a9ee09310b5adcbe (127.2.248.134:33609): Couldn't send request to peer c284970db26b4c95a9ee09310b5adcbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8dbcd048e5a94589a1ea17c449688af4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.072633  6671 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:37.074939  6799 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240501 08:11:37.078689  8403 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240501 08:11:37.090668  8487 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:37.098327  6401 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index -1 to 58, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:37.099552  6799 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
W20240501 08:11:37.138985  6607 consensus_peers.cc:485] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 -> Peer fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129): Couldn't send request to peer fcfccbafc3254a87896f0e72c8078ab5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 217060ad409b4492a7d4ce043578aa49. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.184329  6670 consensus_queue.cc:237] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.46, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:37.186594  6801 raft_consensus.cc:1261] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240501 08:11:37.197736  8498 consensus_queue.cc:1035] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20240501 08:11:37.207727  8487 raft_consensus.cc:2932] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } } }
I20240501 08:11:37.210835  6801 raft_consensus.cc:2932] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } } }
I20240501 08:11:37.213363  6401 catalog_manager.cc:5484] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index -1 to 47, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:37.257275  6608 consensus_peers.cc:485] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97da0f9e56f14f6a985b6ffe8c44b68c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.285254  8553 ts_tablet_manager.cc:902] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Initiating tablet copy from peer 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:37.312526  6541 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:37.317869  6671 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 08:11:37.324607  8371 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
W20240501 08:11:37.329092  6478 consensus_peers.cc:485] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d -> Peer ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997): Couldn't send request to peer ad679e52ba5643f2b11838d14696d1ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cc205621c874d5ab791bb3329988ccc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.332469  7804 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:37.335300  6669 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:37.340905  6398 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index -1 to 75, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:37.350770  8553 tablet_copy_client.cc:308] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:43453
I20240501 08:11:37.385165  6822 tablet_copy_service.cc:140] P 7fb652773c524f639dba8c189b7743ba: Received BeginTabletCopySession request for tablet 74cea6ae2bd446ce8adc7442dea7f030 from peer ad679e52ba5643f2b11838d14696d1ed ({username='slave'} at 127.2.248.135:55093)
I20240501 08:11:37.385495  6822 tablet_copy_service.cc:161] P 7fb652773c524f639dba8c189b7743ba: Beginning new tablet copy session on tablet 74cea6ae2bd446ce8adc7442dea7f030 from peer ad679e52ba5643f2b11838d14696d1ed at {username='slave'} at 127.2.248.135:55093: session id = ad679e52ba5643f2b11838d14696d1ed-74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:11:37.395807  6822 tablet_copy_source_session.cc:205] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:37.407205  8553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cea6ae2bd446ce8adc7442dea7f030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.420989  6799 consensus_queue.cc:237] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:37.439137  8553 tablet_copy_client.cc:783] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:37.439841  8553 tablet_copy_client.cc:647] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:37.444701  6540 raft_consensus.cc:1261] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 08:11:37.446730  8553 tablet_copy_client.cc:523] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:37.446863  8571 ts_tablet_manager.cc:902] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: Initiating tablet copy from peer d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:37.447357  8415 consensus_queue.cc:1035] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
W20240501 08:11:37.452719  6737 consensus_peers.cc:485] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba -> Peer fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129): Couldn't send request to peer fcfccbafc3254a87896f0e72c8078ab5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc900308d554b08b5e54974c0ae77e9. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.459903  8467 raft_consensus.cc:2932] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:37.465626  8553 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap starting.
I20240501 08:11:37.466166  6540 raft_consensus.cc:2932] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index -1 to 75, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:37.478647  6398 catalog_manager.cc:5484] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index -1 to 75, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:37.571202  8571 tablet_copy_client.cc:308] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45487
I20240501 08:11:37.595086  6692 tablet_copy_service.cc:140] P d143a9ab17334a6081de63802bdce654: Received BeginTabletCopySession request for tablet 8dbcd048e5a94589a1ea17c449688af4 from peer c284970db26b4c95a9ee09310b5adcbe ({username='slave'} at 127.2.248.134:47593)
I20240501 08:11:37.595427  6692 tablet_copy_service.cc:161] P d143a9ab17334a6081de63802bdce654: Beginning new tablet copy session on tablet 8dbcd048e5a94589a1ea17c449688af4 from peer c284970db26b4c95a9ee09310b5adcbe at {username='slave'} at 127.2.248.134:47593: session id = c284970db26b4c95a9ee09310b5adcbe-8dbcd048e5a94589a1ea17c449688af4
I20240501 08:11:37.600116  8553 log.cc:824] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:37.603152  6692 tablet_copy_source_session.cc:205] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:37.611265  8571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbcd048e5a94589a1ea17c449688af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.641242  8571 tablet_copy_client.cc:783] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:37.641681  8571 tablet_copy_client.cc:647] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:37.645288  8571 tablet_copy_client.cc:523] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:37.649713  8571 tablet_bootstrap.cc:492] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap starting.
I20240501 08:11:37.673619  8587 ts_tablet_manager.cc:902] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: Initiating tablet copy from peer d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:37.692770  6670 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
W20240501 08:11:37.715034  6608 consensus_peers.cc:485] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 -> Peer c284970db26b4c95a9ee09310b5adcbe (127.2.248.134:33609): Couldn't send request to peer c284970db26b4c95a9ee09310b5adcbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3beb371e6b749ec9580bca95f0908d1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.721592  8571 log.cc:824] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:37.729364  6799 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240501 08:11:37.740489  8403 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 08:11:37.749590  8591 ts_tablet_manager.cc:902] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: Initiating tablet copy from peer d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:37.759233  8594 ts_tablet_manager.cc:902] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:37.772205  7733 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:37.778952  6401 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index -1 to 77, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:37.793503  6801 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:37.795482  8553 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=684 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:37.796053  8553 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap complete.
I20240501 08:11:37.796547  8553 ts_tablet_manager.cc:1367] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Time spent bootstrapping tablet: real 0.331s	user 0.111s	sys 0.012s
I20240501 08:11:37.801271  8594 tablet_copy_client.cc:308] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:37.802131  8591 tablet_copy_client.cc:308] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45487
I20240501 08:11:37.809785  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet 9cc205621c874d5ab791bb3329988ccc from peer ad679e52ba5643f2b11838d14696d1ed ({username='slave'} at 127.2.248.135:55761)
I20240501 08:11:37.810034  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet 9cc205621c874d5ab791bb3329988ccc from peer ad679e52ba5643f2b11838d14696d1ed at {username='slave'} at 127.2.248.135:55761: session id = ad679e52ba5643f2b11838d14696d1ed-9cc205621c874d5ab791bb3329988ccc
I20240501 08:11:37.813786  6562 tablet_copy_source_session.cc:205] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:37.815820  8594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc205621c874d5ab791bb3329988ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.826673  8594 tablet_copy_client.cc:783] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:37.826926  8594 tablet_copy_client.cc:647] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:37.826138  8553 raft_consensus.cc:348] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:37.830207  8553 raft_consensus.cc:729] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad679e52ba5643f2b11838d14696d1ed, State: Initialized, Role: LEARNER
I20240501 08:11:37.832188  8594 tablet_copy_client.cc:523] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:37.838999  8594 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: Bootstrap starting.
I20240501 08:11:37.841303  8553 consensus_queue.cc:260] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:37.849227  8571 tablet_bootstrap.cc:492] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:37.850004  8571 tablet_bootstrap.cc:492] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap complete.
I20240501 08:11:37.850551  8571 ts_tablet_manager.cc:1367] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: Time spent bootstrapping tablet: real 0.201s	user 0.112s	sys 0.008s
I20240501 08:11:37.856840  8571 raft_consensus.cc:348] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:37.857383  8571 raft_consensus.cc:729] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c284970db26b4c95a9ee09310b5adcbe, State: Initialized, Role: LEARNER
I20240501 08:11:37.858002  8571 consensus_queue.cc:260] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:37.865274  6540 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
W20240501 08:11:37.867339  6478 consensus_peers.cc:485] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527072daf1ea4b478a1b35f4a5c4bf3c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:37.867472  6801 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240501 08:11:37.868162  8371 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 08:11:37.869217  8553 ts_tablet_manager.cc:1398] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Time spent starting tablet: real 0.072s	user 0.009s	sys 0.000s
I20240501 08:11:37.870337  6822 tablet_copy_service.cc:342] P 7fb652773c524f639dba8c189b7743ba: Request end of tablet copy session ad679e52ba5643f2b11838d14696d1ed-74cea6ae2bd446ce8adc7442dea7f030 received from {username='slave'} at 127.2.248.135:55093
I20240501 08:11:37.870512  6822 tablet_copy_service.cc:433] P 7fb652773c524f639dba8c189b7743ba: ending tablet copy session ad679e52ba5643f2b11838d14696d1ed-74cea6ae2bd446ce8adc7442dea7f030 on tablet 74cea6ae2bd446ce8adc7442dea7f030 with peer ad679e52ba5643f2b11838d14696d1ed
I20240501 08:11:37.874359  8571 ts_tablet_manager.cc:1398] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.007s
I20240501 08:11:37.875304  6692 tablet_copy_service.cc:342] P d143a9ab17334a6081de63802bdce654: Request end of tablet copy session c284970db26b4c95a9ee09310b5adcbe-8dbcd048e5a94589a1ea17c449688af4 received from {username='slave'} at 127.2.248.134:47593
I20240501 08:11:37.875473  6692 tablet_copy_service.cc:433] P d143a9ab17334a6081de63802bdce654: ending tablet copy session c284970db26b4c95a9ee09310b5adcbe-8dbcd048e5a94589a1ea17c449688af4 on tablet 8dbcd048e5a94589a1ea17c449688af4 with peer c284970db26b4c95a9ee09310b5adcbe
I20240501 08:11:37.879165  8371 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } } }
I20240501 08:11:37.884291  6398 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index -1 to 64, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:37.889698  8138 raft_consensus.cc:1203] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.61]   Dedup: 1.56->[1.57-1.61]
I20240501 08:11:37.894767  6801 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } } }
I20240501 08:11:37.911332  6692 tablet_copy_service.cc:140] P d143a9ab17334a6081de63802bdce654: Received BeginTabletCopySession request for tablet 97da0f9e56f14f6a985b6ffe8c44b68c from peer 0257c0e8038d4e3fa9580cb47278f16f ({username='slave'} at 127.2.248.133:49109)
I20240501 08:11:37.911525  6692 tablet_copy_service.cc:161] P d143a9ab17334a6081de63802bdce654: Beginning new tablet copy session on tablet 97da0f9e56f14f6a985b6ffe8c44b68c from peer 0257c0e8038d4e3fa9580cb47278f16f at {username='slave'} at 127.2.248.133:49109: session id = 0257c0e8038d4e3fa9580cb47278f16f-97da0f9e56f14f6a985b6ffe8c44b68c
I20240501 08:11:37.915323  6692 tablet_copy_source_session.cc:205] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:37.935369  8591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97da0f9e56f14f6a985b6ffe8c44b68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.960804  8591 tablet_copy_client.cc:783] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:37.961553  8591 tablet_copy_client.cc:647] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:37.969647  8616 ts_tablet_manager.cc:902] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: Initiating tablet copy from peer 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:37.979548  8591 tablet_copy_client.cc:523] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:37.990942  8591 tablet_bootstrap.cc:492] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap starting.
I20240501 08:11:37.998543  8403 raft_consensus.cc:1050] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654: attempting to promote NON_VOTER c284970db26b4c95a9ee09310b5adcbe to VOTER
I20240501 08:11:37.999368  8403 consensus_queue.cc:237] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:38.002247  6541 raft_consensus.cc:1261] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240501 08:11:38.002882  7733 consensus_queue.cc:1035] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 08:11:38.005185  8137 raft_consensus.cc:1261] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240501 08:11:38.005137  7733 raft_consensus.cc:2932] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index 44 to 64, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:38.008222  8498 consensus_queue.cc:1035] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 08:11:38.011761  6541 raft_consensus.cc:2932] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 44 to 64, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:38.013413  8138 raft_consensus.cc:2932] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 44 to 64, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:38.013818  6401 catalog_manager.cc:5484] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 44 to 64, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:38.033162  8616 tablet_copy_client.cc:308] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:43453
I20240501 08:11:38.042084  8587 tablet_copy_client.cc:308] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45487
I20240501 08:11:38.063947  6822 tablet_copy_service.cc:140] P 7fb652773c524f639dba8c189b7743ba: Received BeginTabletCopySession request for tablet 8cc900308d554b08b5e54974c0ae77e9 from peer fcfccbafc3254a87896f0e72c8078ab5 ({username='slave'} at 127.2.248.132:56025)
I20240501 08:11:38.064145  6822 tablet_copy_service.cc:161] P 7fb652773c524f639dba8c189b7743ba: Beginning new tablet copy session on tablet 8cc900308d554b08b5e54974c0ae77e9 from peer fcfccbafc3254a87896f0e72c8078ab5 at {username='slave'} at 127.2.248.132:56025: session id = fcfccbafc3254a87896f0e72c8078ab5-8cc900308d554b08b5e54974c0ae77e9
I20240501 08:11:38.067029  6692 tablet_copy_service.cc:140] P d143a9ab17334a6081de63802bdce654: Received BeginTabletCopySession request for tablet 217060ad409b4492a7d4ce043578aa49 from peer fcfccbafc3254a87896f0e72c8078ab5 ({username='slave'} at 127.2.248.132:55095)
I20240501 08:11:38.067166  6692 tablet_copy_service.cc:161] P d143a9ab17334a6081de63802bdce654: Beginning new tablet copy session on tablet 217060ad409b4492a7d4ce043578aa49 from peer fcfccbafc3254a87896f0e72c8078ab5 at {username='slave'} at 127.2.248.132:55095: session id = fcfccbafc3254a87896f0e72c8078ab5-217060ad409b4492a7d4ce043578aa49
I20240501 08:11:38.068490  6822 tablet_copy_source_session.cc:205] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:38.069926  6692 tablet_copy_source_session.cc:205] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:38.074105  8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217060ad409b4492a7d4ce043578aa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:38.076771  8616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc900308d554b08b5e54974c0ae77e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:38.090689  6671 consensus_queue.cc:237] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:38.091935  8587 tablet_copy_client.cc:783] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:38.092305  8587 tablet_copy_client.cc:647] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:38.095116  8587 tablet_copy_client.cc:523] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:38.095711  8616 tablet_copy_client.cc:783] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:38.095957  8616 tablet_copy_client.cc:647] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:38.100407  8138 raft_consensus.cc:1261] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240501 08:11:38.101330  8498 consensus_queue.cc:1035] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240501 08:11:38.104646  8498 raft_consensus.cc:2932] T 8dbcd048e5a94589a1ea17c449688af4 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:38.104961  8616 tablet_copy_client.cc:523] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:38.108484  8137 raft_consensus.cc:2932] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:38.108915  8616 tablet_bootstrap.cc:492] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap starting.
I20240501 08:11:38.115119  8587 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap starting.
I20240501 08:11:38.116834  6541 consensus_queue.cc:237] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:38.118888  6398 catalog_manager.cc:5484] T 8dbcd048e5a94589a1ea17c449688af4 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: config changed from index 64 to 66, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:38.119653  8594 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=82 overwritten=0 applied=81 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:38.120165  8594 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: Bootstrap complete.
I20240501 08:11:38.120438  8594 ts_tablet_manager.cc:1367] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: Time spent bootstrapping tablet: real 0.282s	user 0.114s	sys 0.023s
I20240501 08:11:38.121518  8594 raft_consensus.cc:348] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:38.124884  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8dbcd048e5a94589a1ea17c449688af4 with cas_config_opid_index 64: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:38.123034  6522 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbcd048e5a94589a1ea17c449688af4 with delete_type TABLET_DATA_TOMBSTONED (TS b0190f14862148f4bde78afbbf1ba43d not found in new config with opid_index 66) from {username='slave'} at 127.0.0.1:51260
W20240501 08:11:38.134243  6478 consensus_peers.cc:485] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d -> Peer ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997): Couldn't send request to peer ad679e52ba5643f2b11838d14696d1ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f476973a1f24e09b30075aff39ce31c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:38.137516  6669 raft_consensus.cc:1261] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 08:11:38.141757  8631 tablet_replica.cc:330] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: stopping tablet replica
I20240501 08:11:38.142349  8631 raft_consensus.cc:2229] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:38.142825  8631 raft_consensus.cc:2258] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:38.143805  7435 consensus_queue.cc:1035] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 08:11:38.145217  8631 ts_tablet_manager.cc:1875] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:38.152979  8594 raft_consensus.cc:729] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad679e52ba5643f2b11838d14696d1ed, State: Initialized, Role: LEARNER
I20240501 08:11:38.153718  8594 consensus_queue.cc:260] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 81, Last appended: 1.82, Last appended by leader: 82, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:38.154404  8594 ts_tablet_manager.cc:1398] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.004s
I20240501 08:11:38.155794  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session ad679e52ba5643f2b11838d14696d1ed-9cc205621c874d5ab791bb3329988ccc received from {username='slave'} at 127.2.248.135:55761
I20240501 08:11:38.155952  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session ad679e52ba5643f2b11838d14696d1ed-9cc205621c874d5ab791bb3329988ccc on tablet 9cc205621c874d5ab791bb3329988ccc with peer ad679e52ba5643f2b11838d14696d1ed
I20240501 08:11:38.156297  8392 raft_consensus.cc:2932] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:38.167006  6669 raft_consensus.cc:2932] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:38.169071  6401 catalog_manager.cc:5484] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index -1 to 78, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:38.185060  8631 ts_tablet_manager.cc:1888] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.65
I20240501 08:11:38.186875  8631 log.cc:1196] T 8dbcd048e5a94589a1ea17c449688af4 P b0190f14862148f4bde78afbbf1ba43d: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/8dbcd048e5a94589a1ea17c449688af4
I20240501 08:11:38.183925  8587 log.cc:824] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:38.187991  6385 catalog_manager.cc:4830] TS b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709): tablet 8dbcd048e5a94589a1ea17c449688af4 (table rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]) successfully deleted
I20240501 08:11:38.190639  8275 raft_consensus.cc:1203] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.66]   Dedup: 1.53->[1.54-1.66]
I20240501 08:11:38.264493  8643 ts_tablet_manager.cc:902] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:38.292838  7641 raft_consensus.cc:1050] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: attempting to promote NON_VOTER ad679e52ba5643f2b11838d14696d1ed to VOTER
I20240501 08:11:38.293884  7641 consensus_queue.cc:237] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:38.297243  8275 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240501 08:11:38.298939  8577 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 66, Time since last communication: 0.000s
I20240501 08:11:38.302985  8577 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index 44 to 68, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.308969  8275 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 44 to 68, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.321556  8571 ts_tablet_manager.cc:902] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: Initiating tablet copy from peer d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:38.322777  6401 catalog_manager.cc:5484] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 44 to 68, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:38.335367  6801 consensus_queue.cc:237] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:38.338665  6670 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240501 08:11:38.367660  8275 raft_consensus.cc:1203] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.74->[1.75-1.85]   Dedup: 1.82->[1.83-1.85]
I20240501 08:11:38.394268  8571 tablet_copy_client.cc:308] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45487
I20240501 08:11:38.394984  6692 tablet_copy_service.cc:140] P d143a9ab17334a6081de63802bdce654: Received BeginTabletCopySession request for tablet a3beb371e6b749ec9580bca95f0908d1 from peer c284970db26b4c95a9ee09310b5adcbe ({username='slave'} at 127.2.248.134:47593)
I20240501 08:11:38.395166  6692 tablet_copy_service.cc:161] P d143a9ab17334a6081de63802bdce654: Beginning new tablet copy session on tablet a3beb371e6b749ec9580bca95f0908d1 from peer c284970db26b4c95a9ee09310b5adcbe at {username='slave'} at 127.2.248.134:47593: session id = c284970db26b4c95a9ee09310b5adcbe-a3beb371e6b749ec9580bca95f0908d1
I20240501 08:11:38.396023  8275 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240501 08:11:38.398020  6692 tablet_copy_source_session.cc:205] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:38.399252  7641 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240501 08:11:38.399699  8571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3beb371e6b749ec9580bca95f0908d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:38.404649  8415 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 68 to 70, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.408958  8275 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 68 to 70, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.409368  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74cea6ae2bd446ce8adc7442dea7f030 with cas_config_opid_index 68: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:38.411635  6398 catalog_manager.cc:5484] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 68 to 70, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:38.413246  8571 tablet_copy_client.cc:783] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:38.413491  8571 tablet_copy_client.cc:647] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:38.417120  8571 tablet_copy_client.cc:523] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:38.417271  6652 tablet_service.cc:1514] Processing DeleteTablet for tablet 74cea6ae2bd446ce8adc7442dea7f030 with delete_type TABLET_DATA_TOMBSTONED (TS d143a9ab17334a6081de63802bdce654 not found in new config with opid_index 70) from {username='slave'} at 127.0.0.1:54172
I20240501 08:11:38.420954  8571 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap starting.
I20240501 08:11:38.433116  8651 tablet_replica.cc:330] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: stopping tablet replica
I20240501 08:11:38.434896  8651 raft_consensus.cc:2229] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:38.436446  8651 pending_rounds.cc:62] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Trying to abort 1 pending ops.
I20240501 08:11:38.437064  8651 pending_rounds.cc:68] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Aborting op as it isn't in flight: id { term: 1 index: 67 } timestamp: 7022801298583568384 op_type: WRITE_OP write_request { tablet_id: "74cea6ae2bd446ce8adc7442dea7f030" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000-\331\274k\332\221\306\237\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""bbb2598974289ffa" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022801298395041792 authz_token { token_data: "\010\342\364\307\261\006\"3\n\005slave\022*\n 39aff6d5829146fca6483f16b903dc71\020\001\030\001 \001(\001" signature: "<redacted>""KXQ($\206\202\3706\0160\036\016\223)\316C\205)N_\210J@\320\222\313x\"\334\304\214\304l\315q+h\021\332F`\013^U\340\027\257\216\355e\313\005&\232r\017\320)%\r\333\270\226" signing_key_seq_num: 0 } } request_id { client_id: "abe2406488894ffe911264ca47531ddc" seq_no: 183 first_incomplete_seq_no: 183 attempt_no: 0 }
I20240501 08:11:38.439081  8651 raft_consensus.cc:2258] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:38.444782  8587 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=755 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:38.445389  8587 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap complete.
I20240501 08:11:38.445842  8587 ts_tablet_manager.cc:1367] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: Time spent bootstrapping tablet: real 0.331s	user 0.131s	sys 0.008s
I20240501 08:11:38.445858  8651 ts_tablet_manager.cc:1875] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:38.456763  8587 raft_consensus.cc:348] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:38.457278  8587 raft_consensus.cc:729] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fcfccbafc3254a87896f0e72c8078ab5, State: Initialized, Role: LEARNER
I20240501 08:11:38.457840  8587 consensus_queue.cc:260] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:38.467818  8587 ts_tablet_manager.cc:1398] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5: Time spent starting tablet: real 0.022s	user 0.010s	sys 0.000s
I20240501 08:11:38.468786  6692 tablet_copy_service.cc:342] P d143a9ab17334a6081de63802bdce654: Request end of tablet copy session fcfccbafc3254a87896f0e72c8078ab5-217060ad409b4492a7d4ce043578aa49 received from {username='slave'} at 127.2.248.132:55095
I20240501 08:11:38.468926  6692 tablet_copy_service.cc:433] P d143a9ab17334a6081de63802bdce654: ending tablet copy session fcfccbafc3254a87896f0e72c8078ab5-217060ad409b4492a7d4ce043578aa49 on tablet 217060ad409b4492a7d4ce043578aa49 with peer fcfccbafc3254a87896f0e72c8078ab5
I20240501 08:11:38.469484  8651 ts_tablet_manager.cc:1888] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.67
I20240501 08:11:38.469996  8651 log.cc:1196] T 74cea6ae2bd446ce8adc7442dea7f030 P d143a9ab17334a6081de63802bdce654: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:11:38.472585  6386 catalog_manager.cc:4830] TS d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487): tablet 74cea6ae2bd446ce8adc7442dea7f030 (table rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]) successfully deleted
I20240501 08:11:38.486472  8616 tablet_bootstrap.cc:492] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=82 overwritten=0 applied=82 ignored=0} inserts{seen=767 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:38.486987  8616 tablet_bootstrap.cc:492] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap complete.
I20240501 08:11:38.487288  8616 ts_tablet_manager.cc:1367] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: Time spent bootstrapping tablet: real 0.379s	user 0.139s	sys 0.018s
I20240501 08:11:38.488394  8616 raft_consensus.cc:348] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:38.488698  8616 raft_consensus.cc:729] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fcfccbafc3254a87896f0e72c8078ab5, State: Initialized, Role: LEARNER
I20240501 08:11:38.488919  8616 consensus_queue.cc:260] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.82, Last appended by leader: 82, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:38.489547  8616 ts_tablet_manager.cc:1398] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:11:38.490181  6822 tablet_copy_service.cc:342] P 7fb652773c524f639dba8c189b7743ba: Request end of tablet copy session fcfccbafc3254a87896f0e72c8078ab5-8cc900308d554b08b5e54974c0ae77e9 received from {username='slave'} at 127.2.248.132:56025
I20240501 08:11:38.490295  6822 tablet_copy_service.cc:433] P 7fb652773c524f639dba8c189b7743ba: ending tablet copy session fcfccbafc3254a87896f0e72c8078ab5-8cc900308d554b08b5e54974c0ae77e9 on tablet 8cc900308d554b08b5e54974c0ae77e9 with peer fcfccbafc3254a87896f0e72c8078ab5
I20240501 08:11:38.522583  8594 ts_tablet_manager.cc:902] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:38.533959  8371 raft_consensus.cc:1050] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER ad679e52ba5643f2b11838d14696d1ed to VOTER
I20240501 08:11:38.534981  8371 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:38.537285  8275 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240501 08:11:38.538950  8392 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240501 08:11:38.544333  8371 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index 75 to 86, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.552501  6401 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 75 to 86, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:38.554026  8275 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 75 to 86, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.561312  6541 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:38.565443  8275 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240501 08:11:38.567499  7804 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 86, Time since last communication: 0.000s
I20240501 08:11:38.569651  8594 tablet_copy_client.cc:308] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:38.570250  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet 4f476973a1f24e09b30075aff39ce31c from peer ad679e52ba5643f2b11838d14696d1ed ({username='slave'} at 127.2.248.135:55761)
I20240501 08:11:38.570415  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet 4f476973a1f24e09b30075aff39ce31c from peer ad679e52ba5643f2b11838d14696d1ed at {username='slave'} at 127.2.248.135:55761: session id = ad679e52ba5643f2b11838d14696d1ed-4f476973a1f24e09b30075aff39ce31c
I20240501 08:11:38.572072  7804 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 86 to 88, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.573194  6562 tablet_copy_source_session.cc:205] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:38.574487  8594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f476973a1f24e09b30075aff39ce31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:38.577647  7874 raft_consensus.cc:1203] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.57->[1.58-1.71]   Dedup: 1.70->[1.71-1.71]
I20240501 08:11:38.579715  6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9cc205621c874d5ab791bb3329988ccc with cas_config_opid_index 86: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:38.581971  6398 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 86 to 88, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:38.586867  6652 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cc205621c874d5ab791bb3329988ccc with delete_type TABLET_DATA_TOMBSTONED (TS d143a9ab17334a6081de63802bdce654 not found in new config with opid_index 88) from {username='slave'} at 127.0.0.1:54172
I20240501 08:11:38.587186  8651 tablet_replica.cc:330] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: stopping tablet replica
I20240501 08:11:38.587407  8651 raft_consensus.cc:2229] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:38.587628  8651 raft_consensus.cc:2258] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:38.589409  8651 ts_tablet_manager.cc:1875] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:38.595844  8275 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 86 to 88, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:38.597415  8651 ts_tablet_manager.cc:1888] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.85
I20240501 08:11:38.597585  8651 log.cc:1196] T 9cc205621c874d5ab791bb3329988ccc P d143a9ab17334a6081de63802bdce654: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/9cc205621c874d5ab791bb3329988ccc
I20240501 08:11:38.598328  6386 catalog_manager.cc:4830] TS d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487): tablet 9cc205621c874d5ab791bb3329988ccc (table rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]) successfully deleted
I20240501 08:11:38.605993  8594 tablet_copy_client.cc:783] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:38.608906  8594 tablet_copy_client.cc:647] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:38.611120  8594 tablet_copy_client.cc:523] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:38.614563  8594 tablet_bootstrap.cc:492] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: Bootstrap starting.
I20240501 08:11:38.635808  6541 consensus_queue.cc:237] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:38.658174  8007 raft_consensus.cc:1261] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:38.658965  7435 consensus_queue.cc:1035] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:11:38.659526  6478 consensus_peers.cc:485] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d -> Peer ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997): Couldn't send request to peer ad679e52ba5643f2b11838d14696d1ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b421e60e689c4adfabadb47bd9a9e167. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:38.663796  8403 raft_consensus.cc:1050] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: attempting to promote NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 to VOTER
I20240501 08:11:38.664717  8403 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:38.667050  7873 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 08:11:38.667835  8551 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 08:11:38.670571  8498 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 58 to 74, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.671257  7873 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 58 to 74, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.674444  6799 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 08:11:38.675315  8643 tablet_copy_client.cc:308] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:38.675578  6401 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 58 to 74, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:38.680328  7435 raft_consensus.cc:2932] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:38.686553  8007 raft_consensus.cc:2932] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:38.690129  8403 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 08:11:38.690758  6670 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:38.692757  6799 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 58 to 74, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.693672  7873 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240501 08:11:38.694489  8403 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240501 08:11:38.697469  8551 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
W20240501 08:11:38.698103  6476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.130:45487, user_credentials={real_user=slave}} blocked reactor thread for 72235.9us
I20240501 08:11:38.698688  6398 catalog_manager.cc:5484] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index -1 to 2, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:38.701109  7873 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 74 to 75, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.703931  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet 527072daf1ea4b478a1b35f4a5c4bf3c from peer 0257c0e8038d4e3fa9580cb47278f16f ({username='slave'} at 127.2.248.133:42855)
I20240501 08:11:38.704063  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet 527072daf1ea4b478a1b35f4a5c4bf3c from peer 0257c0e8038d4e3fa9580cb47278f16f at {username='slave'} at 127.2.248.133:42855: session id = 0257c0e8038d4e3fa9580cb47278f16f-527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:11:38.705093  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 217060ad409b4492a7d4ce043578aa49 with cas_config_opid_index 74: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:38.706480  6401 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 74 to 75, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:38.706791  6562 tablet_copy_source_session.cc:205] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:38.708395  8643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527072daf1ea4b478a1b35f4a5c4bf3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:38.709651  8571 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=771 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:38.710157  8571 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap complete.
I20240501 08:11:38.710670  6781 tablet_service.cc:1514] Processing DeleteTablet for tablet 217060ad409b4492a7d4ce043578aa49 with delete_type TABLET_DATA_TOMBSTONED (TS 7fb652773c524f639dba8c189b7743ba not found in new config with opid_index 75) from {username='slave'} at 127.0.0.1:44490
I20240501 08:11:38.716634  8643 tablet_copy_client.cc:783] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:38.716925  8643 tablet_copy_client.cc:647] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:38.716922  8571 ts_tablet_manager.cc:1367] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: Time spent bootstrapping tablet: real 0.296s	user 0.159s	sys 0.008s
I20240501 08:11:38.718897  8571 raft_consensus.cc:348] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:38.719810  8571 raft_consensus.cc:729] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c284970db26b4c95a9ee09310b5adcbe, State: Initialized, Role: LEARNER
I20240501 08:11:38.719889  8643 tablet_copy_client.cc:523] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:38.720304  8571 consensus_queue.cc:260] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:38.723079  8643 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap starting.
I20240501 08:11:38.729190  8571 ts_tablet_manager.cc:1398] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I20240501 08:11:38.730391  6692 tablet_copy_service.cc:342] P d143a9ab17334a6081de63802bdce654: Request end of tablet copy session c284970db26b4c95a9ee09310b5adcbe-a3beb371e6b749ec9580bca95f0908d1 received from {username='slave'} at 127.2.248.134:47593
I20240501 08:11:38.730501  6692 tablet_copy_service.cc:433] P d143a9ab17334a6081de63802bdce654: ending tablet copy session c284970db26b4c95a9ee09310b5adcbe-a3beb371e6b749ec9580bca95f0908d1 on tablet a3beb371e6b749ec9580bca95f0908d1 with peer c284970db26b4c95a9ee09310b5adcbe
I20240501 08:11:38.739156  8591 tablet_bootstrap.cc:492] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=60 ignored=0} inserts{seen=745 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:38.739657  8591 tablet_bootstrap.cc:492] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap complete.
I20240501 08:11:38.739943  8591 ts_tablet_manager.cc:1367] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: Time spent bootstrapping tablet: real 0.749s	user 0.102s	sys 0.020s
I20240501 08:11:38.741011  8591 raft_consensus.cc:348] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:38.748903  8673 tablet_replica.cc:330] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: stopping tablet replica
I20240501 08:11:38.749284  8673 raft_consensus.cc:2229] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:38.749807  8673 raft_consensus.cc:2258] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:38.751679  8673 ts_tablet_manager.cc:1875] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:38.761368  8673 ts_tablet_manager.cc:1888] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74
I20240501 08:11:38.761539  8138 raft_consensus.cc:1203] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.76->[1.77-1.89]   Dedup: 1.86->[1.87-1.89]
I20240501 08:11:38.761655  8591 raft_consensus.cc:729] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0257c0e8038d4e3fa9580cb47278f16f, State: Initialized, Role: LEARNER
I20240501 08:11:38.761889  8591 consensus_queue.cc:260] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:38.762529  8591 ts_tablet_manager.cc:1398] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.006s
I20240501 08:11:38.763499  8673 log.cc:1196] T 217060ad409b4492a7d4ce043578aa49 P 7fb652773c524f639dba8c189b7743ba: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/217060ad409b4492a7d4ce043578aa49
I20240501 08:11:38.764478  6386 catalog_manager.cc:4830] TS 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453): tablet 217060ad409b4492a7d4ce043578aa49 (table rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]) successfully deleted
I20240501 08:11:38.777534  6692 tablet_copy_service.cc:342] P d143a9ab17334a6081de63802bdce654: Request end of tablet copy session 0257c0e8038d4e3fa9580cb47278f16f-97da0f9e56f14f6a985b6ffe8c44b68c received from {username='slave'} at 127.2.248.133:49109
I20240501 08:11:38.777658  6692 tablet_copy_service.cc:433] P d143a9ab17334a6081de63802bdce654: ending tablet copy session 0257c0e8038d4e3fa9580cb47278f16f-97da0f9e56f14f6a985b6ffe8c44b68c on tablet 97da0f9e56f14f6a985b6ffe8c44b68c with peer 0257c0e8038d4e3fa9580cb47278f16f
I20240501 08:11:38.789867  6801 consensus_queue.cc:237] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
W20240501 08:11:38.795831  6737 consensus_peers.cc:485] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba -> Peer fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129): Couldn't send request to peer fcfccbafc3254a87896f0e72c8078ab5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5924bdc199d04ba495059ea3c60f2792. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:38.796940  6670 raft_consensus.cc:1261] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240501 08:11:38.798775  7873 raft_consensus.cc:1203] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.74->[1.75-1.86]   Dedup: 1.82->[1.83-1.86]
I20240501 08:11:38.804528  8577 consensus_queue.cc:1035] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240501 08:11:38.807261  8577 raft_consensus.cc:2932] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index -1 to 100, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:38.812716  6671 raft_consensus.cc:2932] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index -1 to 100, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:38.818163  6398 catalog_manager.cc:5484] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index -1 to 100, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:38.862912  7641 raft_consensus.cc:1050] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba: attempting to promote NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 to VOTER
I20240501 08:11:38.863652  7641 consensus_queue.cc:237] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:38.867723  7874 raft_consensus.cc:1261] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 08:11:38.868284  8577 consensus_queue.cc:1035] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240501 08:11:38.873170  8577 raft_consensus.cc:2932] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 75 to 87, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.876780  7874 raft_consensus.cc:2932] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 75 to 87, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.881590  6401 catalog_manager.cc:5484] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 reported cstate change: config changed from index 75 to 87, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.887756  6801 consensus_queue.cc:237] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:38.889797  7874 raft_consensus.cc:1261] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240501 08:11:38.890295  8577 consensus_queue.cc:1035] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240501 08:11:38.893268  8415 raft_consensus.cc:2932] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 87 to 88, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.893940  7873 raft_consensus.cc:2932] T 8cc900308d554b08b5e54974c0ae77e9 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 87 to 88, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:38.897820  8643 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=68 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:38.898406  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cc900308d554b08b5e54974c0ae77e9 with cas_config_opid_index 87: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:38.898280  8643 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap complete.
I20240501 08:11:38.898964  8643 ts_tablet_manager.cc:1367] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: Time spent bootstrapping tablet: real 0.176s	user 0.124s	sys 0.002s
I20240501 08:11:38.899740  6398 catalog_manager.cc:5484] T 8cc900308d554b08b5e54974c0ae77e9 P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 87 to 88, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:38.904086  8643 raft_consensus.cc:348] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:38.904805  8643 raft_consensus.cc:729] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0257c0e8038d4e3fa9580cb47278f16f, State: Initialized, Role: LEARNER
I20240501 08:11:38.905097  8643 consensus_queue.cc:260] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:38.907940  8643 ts_tablet_manager.cc:1398] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.002s
I20240501 08:11:38.908113  6522 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc900308d554b08b5e54974c0ae77e9 with delete_type TABLET_DATA_TOMBSTONED (TS b0190f14862148f4bde78afbbf1ba43d not found in new config with opid_index 88) from {username='slave'} at 127.0.0.1:51260
I20240501 08:11:38.910632  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session 0257c0e8038d4e3fa9580cb47278f16f-527072daf1ea4b478a1b35f4a5c4bf3c received from {username='slave'} at 127.2.248.133:42855
I20240501 08:11:38.910756  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session 0257c0e8038d4e3fa9580cb47278f16f-527072daf1ea4b478a1b35f4a5c4bf3c on tablet 527072daf1ea4b478a1b35f4a5c4bf3c with peer 0257c0e8038d4e3fa9580cb47278f16f
I20240501 08:11:38.915907  6541 raft_consensus.cc:1261] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 08:11:38.939113  8689 tablet_replica.cc:330] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: stopping tablet replica
I20240501 08:11:38.939321  8689 raft_consensus.cc:2229] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:38.939805  8689 raft_consensus.cc:2258] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:38.940912  8689 ts_tablet_manager.cc:1875] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:38.969558  8689 ts_tablet_manager.cc:1888] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.86
I20240501 08:11:38.969713  8689 log.cc:1196] T 8cc900308d554b08b5e54974c0ae77e9 P b0190f14862148f4bde78afbbf1ba43d: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/8cc900308d554b08b5e54974c0ae77e9
I20240501 08:11:38.982407  8498 raft_consensus.cc:1050] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: attempting to promote NON_VOTER c284970db26b4c95a9ee09310b5adcbe to VOTER
I20240501 08:11:38.984078  6385 catalog_manager.cc:4830] TS b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709): tablet 8cc900308d554b08b5e54974c0ae77e9 (table rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]) successfully deleted
I20240501 08:11:38.983407  8498 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:38.986234  8138 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240501 08:11:38.986769  8403 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240501 08:11:38.990089  8138 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 77 to 91, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:38.996694  8403 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index 77 to 91, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:39.002888  6398 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: config changed from index 77 to 91, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:39.009167  6669 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:39.011977  8138 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240501 08:11:39.012786  8403 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 08:11:39.013785  6798 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240501 08:11:39.015122  8403 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index 91 to 92, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:39.020467  8138 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index 91 to 92, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:39.024950  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3beb371e6b749ec9580bca95f0908d1 with cas_config_opid_index 91: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:39.026401  6401 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 91 to 92, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:39.031059  6781 tablet_service.cc:1514] Processing DeleteTablet for tablet a3beb371e6b749ec9580bca95f0908d1 with delete_type TABLET_DATA_TOMBSTONED (TS 7fb652773c524f639dba8c189b7743ba not found in new config with opid_index 92) from {username='slave'} at 127.0.0.1:44490
I20240501 08:11:39.031617  8673 tablet_replica.cc:330] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: stopping tablet replica
I20240501 08:11:39.031818  8673 raft_consensus.cc:2229] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:39.031961  8673 pending_rounds.cc:62] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Trying to abort 1 pending ops.
I20240501 08:11:39.032068  8673 pending_rounds.cc:68] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Aborting op as it isn't in flight: id { term: 1 index: 90 } timestamp: 7022801301414531072 op_type: WRITE_OP write_request { tablet_id: "a3beb371e6b749ec9580bca95f0908d1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\233t?e4G\013\350\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""69f3adcf7a300d1a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022801301208219648 authz_token { token_data: "\010\342\364\307\261\006\"3\n\005slave\022*\n 585c1bb498bf42bea9e424b9373c5233\020\001\030\001 \001(\001" signature: "<redacted>""5t\277m\226v\025\007\367\335\tkw\341gf\302?;\356\202C0ur\327\225z\004\221\335\311\325\3216O9\343#\001r\217\213I\203(>[\021\355\240\025\007?:\335Tc\305\266\271\344\306\345" signing_key_seq_num: 0 } } request_id { client_id: "93243535480b4659b67d21f63ee00a97" seq_no: 264 first_incomplete_seq_no: 264 attempt_no: 0 }
I20240501 08:11:39.032753  8673 raft_consensus.cc:2258] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:39.034559  8673 ts_tablet_manager.cc:1875] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:39.055697  6541 consensus_queue.cc:237] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:39.043885  8673 ts_tablet_manager.cc:1888] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.90
I20240501 08:11:39.063241  8673 log.cc:1196] T a3beb371e6b749ec9580bca95f0908d1 P 7fb652773c524f639dba8c189b7743ba: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3beb371e6b749ec9580bca95f0908d1
I20240501 08:11:39.064249  6386 catalog_manager.cc:4830] TS 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453): tablet a3beb371e6b749ec9580bca95f0908d1 (table rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]) successfully deleted
I20240501 08:11:39.066138  8594 tablet_bootstrap.cc:492] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=83 overwritten=0 applied=82 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:39.066748  8594 tablet_bootstrap.cc:492] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: Bootstrap complete.
I20240501 08:11:39.067059  8594 ts_tablet_manager.cc:1367] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: Time spent bootstrapping tablet: real 0.453s	user 0.129s	sys 0.015s
I20240501 08:11:39.067857  8594 raft_consensus.cc:348] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:39.068874  8594 raft_consensus.cc:729] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad679e52ba5643f2b11838d14696d1ed, State: Initialized, Role: LEARNER
I20240501 08:11:39.069175  8594 consensus_queue.cc:260] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.83, Last appended by leader: 83, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:39.075054  8594 ts_tablet_manager.cc:1398] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240501 08:11:39.076550  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session ad679e52ba5643f2b11838d14696d1ed-4f476973a1f24e09b30075aff39ce31c received from {username='slave'} at 127.2.248.135:55761
I20240501 08:11:39.076706  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session ad679e52ba5643f2b11838d14696d1ed-4f476973a1f24e09b30075aff39ce31c on tablet 4f476973a1f24e09b30075aff39ce31c with peer ad679e52ba5643f2b11838d14696d1ed
I20240501 08:11:39.085172  8594 ts_tablet_manager.cc:902] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:39.090550  6671 raft_consensus.cc:1261] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240501 08:11:39.098986  8371 consensus_queue.cc:1035] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240501 08:11:39.114997  7804 raft_consensus.cc:2932] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:39.116400  6671 raft_consensus.cc:2932] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:39.125934  6398 catalog_manager.cc:5484] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index -1 to 71, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:39.136338  8274 raft_consensus.cc:1203] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.77->[1.78-1.89]   Dedup: 1.83->[1.84-1.89]
I20240501 08:11:39.149556  8007 raft_consensus.cc:1203] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.63->[1.64-1.74]   Dedup: 1.69->[1.70-1.74]
W20240501 08:11:39.151206  6478 consensus_peers.cc:485] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d -> Peer c284970db26b4c95a9ee09310b5adcbe (127.2.248.134:33609): Couldn't send request to peer c284970db26b4c95a9ee09310b5adcbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27f9665553394cf4a53e8ebc82fbc575. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:39.176745  8594 tablet_copy_client.cc:308] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:39.177470  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet b421e60e689c4adfabadb47bd9a9e167 from peer ad679e52ba5643f2b11838d14696d1ed ({username='slave'} at 127.2.248.135:55761)
I20240501 08:11:39.177654  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet b421e60e689c4adfabadb47bd9a9e167 from peer ad679e52ba5643f2b11838d14696d1ed at {username='slave'} at 127.2.248.135:55761: session id = ad679e52ba5643f2b11838d14696d1ed-b421e60e689c4adfabadb47bd9a9e167
I20240501 08:11:39.180617  6562 tablet_copy_source_session.cc:205] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:39.181794  8594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b421e60e689c4adfabadb47bd9a9e167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.201195  8594 tablet_copy_client.cc:783] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:39.201449  8594 tablet_copy_client.cc:647] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:39.221511  8594 tablet_copy_client.cc:523] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:39.224959  8594 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap starting.
I20240501 08:11:39.257824  8594 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:39.260038  8594 tablet_bootstrap.cc:492] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap complete.
I20240501 08:11:39.260705  8594 ts_tablet_manager.cc:1367] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: Time spent bootstrapping tablet: real 0.036s	user 0.004s	sys 0.004s
I20240501 08:11:39.263427  8594 raft_consensus.cc:348] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:39.263653  8594 raft_consensus.cc:729] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad679e52ba5643f2b11838d14696d1ed, State: Initialized, Role: LEARNER
I20240501 08:11:39.263885  8594 consensus_queue.cc:260] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:39.266551  8710 ts_tablet_manager.cc:902] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: Initiating tablet copy from peer 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:39.267379  8594 ts_tablet_manager.cc:1398] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240501 08:11:39.268237  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session ad679e52ba5643f2b11838d14696d1ed-b421e60e689c4adfabadb47bd9a9e167 received from {username='slave'} at 127.2.248.135:55761
I20240501 08:11:39.268361  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session ad679e52ba5643f2b11838d14696d1ed-b421e60e689c4adfabadb47bd9a9e167 on tablet b421e60e689c4adfabadb47bd9a9e167 with peer ad679e52ba5643f2b11838d14696d1ed
I20240501 08:11:39.296242  7435 raft_consensus.cc:1050] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER ad679e52ba5643f2b11838d14696d1ed to VOTER
I20240501 08:11:39.296908  7435 consensus_queue.cc:237] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:39.306710  8275 raft_consensus.cc:1261] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 08:11:39.308140  8700 consensus_queue.cc:1035] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240501 08:11:39.310801  8392 raft_consensus.cc:2932] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 78 to 93, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:39.324074  6401 catalog_manager.cc:5484] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 78 to 93, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:39.342068  8006 raft_consensus.cc:1203] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.46->[1.47-1.70]   Dedup: 1.61->[1.62-1.70]
I20240501 08:11:39.347383  8275 raft_consensus.cc:2932] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 78 to 93, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:39.351816  6541 consensus_queue.cc:237] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:39.369393  6670 raft_consensus.cc:1261] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 08:11:39.446568  8273 raft_consensus.cc:1261] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 08:11:39.462405  8700 consensus_queue.cc:1035] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 08:11:39.474895  8721 ts_tablet_manager.cc:902] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:39.484369  8710 tablet_copy_client.cc:308] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:43453
I20240501 08:11:39.484066  7804 raft_consensus.cc:2932] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:39.485639  6822 tablet_copy_service.cc:140] P 7fb652773c524f639dba8c189b7743ba: Received BeginTabletCopySession request for tablet 5924bdc199d04ba495059ea3c60f2792 from peer fcfccbafc3254a87896f0e72c8078ab5 ({username='slave'} at 127.2.248.132:56025)
I20240501 08:11:39.485787  6822 tablet_copy_service.cc:161] P 7fb652773c524f639dba8c189b7743ba: Beginning new tablet copy session on tablet 5924bdc199d04ba495059ea3c60f2792 from peer fcfccbafc3254a87896f0e72c8078ab5 at {username='slave'} at 127.2.248.132:56025: session id = fcfccbafc3254a87896f0e72c8078ab5-5924bdc199d04ba495059ea3c60f2792
I20240501 08:11:39.488524  6822 tablet_copy_source_session.cc:205] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:39.489310  8275 raft_consensus.cc:1203] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 08:11:39.490002  8710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5924bdc199d04ba495059ea3c60f2792. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.492498  8274 raft_consensus.cc:2932] T 4f476973a1f24e09b30075aff39ce31c P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:39.503140  8710 tablet_copy_client.cc:783] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:39.503477  8710 tablet_copy_client.cc:647] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:39.505980  8710 tablet_copy_client.cc:523] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:39.509326  8710 tablet_bootstrap.cc:492] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap starting.
I20240501 08:11:39.513377  6541 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:39.516093  8392 raft_consensus.cc:1050] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f to VOTER
I20240501 08:11:39.517076  8392 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } }
I20240501 08:11:39.519057  8007 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 08:11:39.519732  8700 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 08:11:39.521225  6801 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 08:11:39.521839  8392 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 08:11:39.523658  6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f476973a1f24e09b30075aff39ce31c with cas_config_opid_index 93: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:39.524724  8700 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 64 to 83, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.527568  6801 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 64 to 83, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.537492  6401 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 64 to 83, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.551810  8006 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 64 to 83, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.554219  6401 catalog_manager.cc:5484] T 4f476973a1f24e09b30075aff39ce31c P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 93 to 95, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:11:39.556607  6478 consensus_peers.cc:485] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d -> Peer 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177): Couldn't send request to peer 0257c0e8038d4e3fa9580cb47278f16f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dff918bc67204080bf1c272c2d950045. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:39.559774  6652 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f476973a1f24e09b30075aff39ce31c with delete_type TABLET_DATA_TOMBSTONED (TS d143a9ab17334a6081de63802bdce654 not found in new config with opid_index 95) from {username='slave'} at 127.0.0.1:54172
I20240501 08:11:39.560982  8551 raft_consensus.cc:1050] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654: attempting to promote NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f to VOTER
I20240501 08:11:39.561564  8551 consensus_queue.cc:237] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } }
I20240501 08:11:39.564615  6800 raft_consensus.cc:1261] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 08:11:39.566864  6801 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240501 08:11:39.567414  8392 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 08:11:39.568037  8403 consensus_queue.cc:1035] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240501 08:11:39.573179  6541 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } }
I20240501 08:11:39.574681  8498 raft_consensus.cc:2932] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 47 to 72, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.574965  8007 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240501 08:11:39.576159  7435 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240501 08:11:39.578629  6799 raft_consensus.cc:2932] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 47 to 72, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.580744  8392 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 83 to 84, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.581022  6398 catalog_manager.cc:5484] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 47 to 72, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:39.582911  7804 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index -1 to 104, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } } }
I20240501 08:11:39.584576  6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 527072daf1ea4b478a1b35f4a5c4bf3c with cas_config_opid_index 83: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:39.585167  6801 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index -1 to 104, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } } }
I20240501 08:11:39.587837  8007 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 83 to 84, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.591149  8006 raft_consensus.cc:1261] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 08:11:39.592296  8551 consensus_queue.cc:1035] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240501 08:11:39.594396  8005 raft_consensus.cc:2932] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 47 to 72, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.594918  6401 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 83 to 84, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:39.599642  6401 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index -1 to 104, NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:39.627619  6670 consensus_queue.cc:237] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } }
I20240501 08:11:39.632356  6781 tablet_service.cc:1514] Processing DeleteTablet for tablet 527072daf1ea4b478a1b35f4a5c4bf3c with delete_type TABLET_DATA_TOMBSTONED (TS 7fb652773c524f639dba8c189b7743ba not found in new config with opid_index 84) from {username='slave'} at 127.0.0.1:44490
I20240501 08:11:39.633317  8729 tablet_replica.cc:330] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: stopping tablet replica
I20240501 08:11:39.633576  8729 raft_consensus.cc:2229] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:39.633958  8729 raft_consensus.cc:2258] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:39.636718  8007 raft_consensus.cc:1261] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 08:11:39.637684  8403 consensus_queue.cc:1035] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 08:11:39.639699  8729 ts_tablet_manager.cc:1875] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:39.640113  8551 raft_consensus.cc:2932] T 97da0f9e56f14f6a985b6ffe8c44b68c P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 72 to 74, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.641275  8005 raft_consensus.cc:2932] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 72 to 74, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.647315  8729 ts_tablet_manager.cc:1888] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.92
I20240501 08:11:39.647491  8729 log.cc:1196] T 4f476973a1f24e09b30075aff39ce31c P d143a9ab17334a6081de63802bdce654: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f476973a1f24e09b30075aff39ce31c
I20240501 08:11:39.648341  6386 catalog_manager.cc:4830] TS d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487): tablet 4f476973a1f24e09b30075aff39ce31c (table rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]) successfully deleted
I20240501 08:11:39.650286  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97da0f9e56f14f6a985b6ffe8c44b68c with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:39.650511  6401 catalog_manager.cc:5484] T 97da0f9e56f14f6a985b6ffe8c44b68c P 0257c0e8038d4e3fa9580cb47278f16f reported cstate change: config changed from index 72 to 74, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:39.657112  8734 tablet_replica.cc:330] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: stopping tablet replica
I20240501 08:11:39.657418  8734 raft_consensus.cc:2229] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:39.657886  8734 raft_consensus.cc:2258] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:39.656057  6782 tablet_service.cc:1514] Processing DeleteTablet for tablet 97da0f9e56f14f6a985b6ffe8c44b68c with delete_type TABLET_DATA_TOMBSTONED (TS 7fb652773c524f639dba8c189b7743ba not found in new config with opid_index 74) from {username='slave'} at 127.0.0.1:44490
I20240501 08:11:39.682375  8734 ts_tablet_manager.cc:1875] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:39.700100  8721 tablet_copy_client.cc:308] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:39.706764  8734 ts_tablet_manager.cc:1888] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.83
I20240501 08:11:39.728240  8734 log.cc:1196] T 527072daf1ea4b478a1b35f4a5c4bf3c P 7fb652773c524f639dba8c189b7743ba: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:11:39.732307  8734 tablet_replica.cc:330] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: stopping tablet replica
I20240501 08:11:39.732590  8734 raft_consensus.cc:2229] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:39.732626  6386 catalog_manager.cc:4830] TS 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453): tablet 527072daf1ea4b478a1b35f4a5c4bf3c (table rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]) successfully deleted
I20240501 08:11:39.736485  8734 raft_consensus.cc:2258] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:39.737692  8734 ts_tablet_manager.cc:1875] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:39.754329  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet 27f9665553394cf4a53e8ebc82fbc575 from peer c284970db26b4c95a9ee09310b5adcbe ({username='slave'} at 127.2.248.134:37963)
I20240501 08:11:39.754540  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet 27f9665553394cf4a53e8ebc82fbc575 from peer c284970db26b4c95a9ee09310b5adcbe at {username='slave'} at 127.2.248.134:37963: session id = c284970db26b4c95a9ee09310b5adcbe-27f9665553394cf4a53e8ebc82fbc575
I20240501 08:11:39.757164  6562 tablet_copy_source_session.cc:205] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:39.763325  8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f9665553394cf4a53e8ebc82fbc575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.771102  8721 tablet_copy_client.cc:783] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:39.771395  8721 tablet_copy_client.cc:647] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:39.773430  8734 ts_tablet_manager.cc:1888] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.73
I20240501 08:11:39.773636  8734 log.cc:1196] T 97da0f9e56f14f6a985b6ffe8c44b68c P 7fb652773c524f639dba8c189b7743ba: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/97da0f9e56f14f6a985b6ffe8c44b68c
I20240501 08:11:39.775554  6386 catalog_manager.cc:4830] TS 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453): tablet 97da0f9e56f14f6a985b6ffe8c44b68c (table rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]) successfully deleted
I20240501 08:11:39.781416  8721 tablet_copy_client.cc:523] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:39.784647  8721 tablet_bootstrap.cc:492] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap starting.
I20240501 08:11:39.851987  8710 tablet_bootstrap.cc:492] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=107 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:39.852600  8710 tablet_bootstrap.cc:492] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap complete.
I20240501 08:11:39.852988  8710 ts_tablet_manager.cc:1367] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: Time spent bootstrapping tablet: real 0.344s	user 0.154s	sys 0.023s
I20240501 08:11:39.853976  8710 raft_consensus.cc:348] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:39.854681  8710 raft_consensus.cc:729] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fcfccbafc3254a87896f0e72c8078ab5, State: Initialized, Role: LEARNER
I20240501 08:11:39.854961  8710 consensus_queue.cc:260] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:39.862380  8710 ts_tablet_manager.cc:1398] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.005s
I20240501 08:11:39.866374  6822 tablet_copy_service.cc:342] P 7fb652773c524f639dba8c189b7743ba: Request end of tablet copy session fcfccbafc3254a87896f0e72c8078ab5-5924bdc199d04ba495059ea3c60f2792 received from {username='slave'} at 127.2.248.132:56025
I20240501 08:11:39.866525  6822 tablet_copy_service.cc:433] P 7fb652773c524f639dba8c189b7743ba: ending tablet copy session fcfccbafc3254a87896f0e72c8078ab5-5924bdc199d04ba495059ea3c60f2792 on tablet 5924bdc199d04ba495059ea3c60f2792 with peer fcfccbafc3254a87896f0e72c8078ab5
I20240501 08:11:39.922088  8700 raft_consensus.cc:1050] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER ad679e52ba5643f2b11838d14696d1ed to VOTER
I20240501 08:11:39.922816  8700 consensus_queue.cc:237] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:39.934330  8271 raft_consensus.cc:1261] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:11:39.934640  8007 raft_consensus.cc:1261] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:11:39.944046  8392 consensus_queue.cc:1035] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:11:39.944566  8392 consensus_queue.cc:1035] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:11:39.977481  8721 tablet_bootstrap.cc:492] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=83 overwritten=0 applied=82 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:39.978174  8721 tablet_bootstrap.cc:492] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap complete.
I20240501 08:11:39.978521  8721 ts_tablet_manager.cc:1367] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: Time spent bootstrapping tablet: real 0.194s	user 0.125s	sys 0.011s
I20240501 08:11:39.979480  8721 raft_consensus.cc:348] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:39.980293  8721 raft_consensus.cc:729] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c284970db26b4c95a9ee09310b5adcbe, State: Initialized, Role: LEARNER
I20240501 08:11:39.980548  8721 consensus_queue.cc:260] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.83, Last appended by leader: 83, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:39.987768  8721 ts_tablet_manager.cc:1398] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20240501 08:11:39.988498  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session c284970db26b4c95a9ee09310b5adcbe-27f9665553394cf4a53e8ebc82fbc575 received from {username='slave'} at 127.2.248.134:37963
I20240501 08:11:39.988642  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session c284970db26b4c95a9ee09310b5adcbe-27f9665553394cf4a53e8ebc82fbc575 on tablet 27f9665553394cf4a53e8ebc82fbc575 with peer c284970db26b4c95a9ee09310b5adcbe
I20240501 08:11:39.995033  8758 ts_tablet_manager.cc:902] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:39.997097  8758 tablet_copy_client.cc:308] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:39.997975  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet dff918bc67204080bf1c272c2d950045 from peer 0257c0e8038d4e3fa9580cb47278f16f ({username='slave'} at 127.2.248.133:42855)
I20240501 08:11:39.998193  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet dff918bc67204080bf1c272c2d950045 from peer 0257c0e8038d4e3fa9580cb47278f16f at {username='slave'} at 127.2.248.133:42855: session id = 0257c0e8038d4e3fa9580cb47278f16f-dff918bc67204080bf1c272c2d950045
I20240501 08:11:40.001302  6562 tablet_copy_source_session.cc:205] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:40.003362  8758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff918bc67204080bf1c272c2d950045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.003677  8700 raft_consensus.cc:2932] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:40.009202  8006 raft_consensus.cc:2932] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:40.009697  8758 tablet_copy_client.cc:783] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:40.009930  8758 tablet_copy_client.cc:647] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:40.015656  6401 catalog_manager.cc:5484] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 2 to 3, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:40.017549  8758 tablet_copy_client.cc:523] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:40.002243  8274 raft_consensus.cc:2932] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:40.024268  8758 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap starting.
I20240501 08:11:40.026513  8138 raft_consensus.cc:1203] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.70->[1.71-1.85]   Dedup: 1.83->[1.84-1.85]
I20240501 08:11:40.163507  7873 raft_consensus.cc:1203] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.99->[1.100-1.116]   Dedup: 1.107->[1.108-1.116]
I20240501 08:11:40.261802  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b421e60e689c4adfabadb47bd9a9e167"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:48018
I20240501 08:11:40.262115  6541 raft_consensus.cc:595] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:40.297077  8671 raft_consensus.cc:1050] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER c284970db26b4c95a9ee09310b5adcbe to VOTER
I20240501 08:11:40.297816  8671 consensus_queue.cc:237] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:40.301156  6670 raft_consensus.cc:1261] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240501 08:11:40.302417  7804 consensus_queue.cc:1035] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 08:11:40.305320  8700 raft_consensus.cc:2932] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index 71 to 92, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:40.308660  6671 raft_consensus.cc:2932] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index 71 to 92, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:40.314054  6398 catalog_manager.cc:5484] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 71 to 92, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:40.341584  8136 raft_consensus.cc:1261] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240501 08:11:40.342420  7804 consensus_queue.cc:1035] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 08:11:40.342895  8415 raft_consensus.cc:1050] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba: attempting to promote NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 to VOTER
I20240501 08:11:40.343616  8415 consensus_queue.cc:237] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.118, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:40.346926  7871 raft_consensus.cc:1261] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240501 08:11:40.347324  6671 raft_consensus.cc:1261] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240501 08:11:40.347676  7641 consensus_queue.cc:1035] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 116, Time since last communication: 0.000s
I20240501 08:11:40.347875  8577 consensus_queue.cc:1035] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240501 08:11:40.351153  7641 raft_consensus.cc:2932] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.119: config changed from index 100 to 119, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:40.354947  7871 raft_consensus.cc:1203] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.117->[1.118-1.119]   Dedup: 1.118->[1.119-1.119]
I20240501 08:11:40.356555  8138 raft_consensus.cc:2932] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index 71 to 92, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:40.359655  7871 raft_consensus.cc:2932] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 100 to 119, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:40.369881  6671 raft_consensus.cc:2932] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 100 to 119, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:40.370690  6397 catalog_manager.cc:5484] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 reported cstate change: config changed from index 100 to 119, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:40.376410  6541 consensus_queue.cc:237] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:40.379545  8136 raft_consensus.cc:1261] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240501 08:11:40.380491  8671 consensus_queue.cc:1035] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 08:11:40.385159  6800 consensus_queue.cc:237] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 1.120, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:40.385787  8671 raft_consensus.cc:2932] T 27f9665553394cf4a53e8ebc82fbc575 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 92 to 94, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:40.388861  8136 raft_consensus.cc:2932] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 92 to 94, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:40.391520  6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27f9665553394cf4a53e8ebc82fbc575 with cas_config_opid_index 92: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:40.393991  6401 catalog_manager.cc:5484] T 27f9665553394cf4a53e8ebc82fbc575 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: config changed from index 92 to 94, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:40.402904  6652 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f9665553394cf4a53e8ebc82fbc575 with delete_type TABLET_DATA_TOMBSTONED (TS d143a9ab17334a6081de63802bdce654 not found in new config with opid_index 94) from {username='slave'} at 127.0.0.1:54172
I20240501 08:11:40.409749  7871 raft_consensus.cc:1261] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240501 08:11:40.410598  7641 consensus_queue.cc:1035] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 119, Time since last communication: 0.000s
I20240501 08:11:40.419976  8769 raft_consensus.cc:2932] T 5924bdc199d04ba495059ea3c60f2792 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.121: config changed from index 119 to 121, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:40.423137  7872 raft_consensus.cc:2932] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 119 to 121, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:40.426206  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5924bdc199d04ba495059ea3c60f2792 with cas_config_opid_index 119: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:40.426390  8784 tablet_replica.cc:330] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: stopping tablet replica
I20240501 08:11:40.426656  8784 raft_consensus.cc:2229] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:40.427430  8784 raft_consensus.cc:2258] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:40.428377  8784 ts_tablet_manager.cc:1875] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:40.430691  6394 catalog_manager.cc:5484] T 5924bdc199d04ba495059ea3c60f2792 P fcfccbafc3254a87896f0e72c8078ab5 reported cstate change: config changed from index 119 to 121, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:40.434301  6651 tablet_service.cc:1514] Processing DeleteTablet for tablet 5924bdc199d04ba495059ea3c60f2792 with delete_type TABLET_DATA_TOMBSTONED (TS d143a9ab17334a6081de63802bdce654 not found in new config with opid_index 121) from {username='slave'} at 127.0.0.1:54172
I20240501 08:11:40.435431  8784 ts_tablet_manager.cc:1888] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.93
I20240501 08:11:40.435546  8784 log.cc:1196] T 27f9665553394cf4a53e8ebc82fbc575 P d143a9ab17334a6081de63802bdce654: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/27f9665553394cf4a53e8ebc82fbc575
I20240501 08:11:40.436452  8784 tablet_replica.cc:330] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: stopping tablet replica
I20240501 08:11:40.436600  6386 catalog_manager.cc:4830] TS d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487): tablet 27f9665553394cf4a53e8ebc82fbc575 (table rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]) successfully deleted
I20240501 08:11:40.436641  8784 raft_consensus.cc:2229] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:40.436902  8784 raft_consensus.cc:2258] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:40.437846  8784 ts_tablet_manager.cc:1875] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:40.443629  8784 ts_tablet_manager.cc:1888] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.119
I20240501 08:11:40.443750  8784 log.cc:1196] T 5924bdc199d04ba495059ea3c60f2792 P d143a9ab17334a6081de63802bdce654: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/5924bdc199d04ba495059ea3c60f2792
I20240501 08:11:40.444862  6386 catalog_manager.cc:4830] TS d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487): tablet 5924bdc199d04ba495059ea3c60f2792 (table rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]) successfully deleted
I20240501 08:11:40.502411  8671 raft_consensus.cc:979] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: : Instructing follower 0257c0e8038d4e3fa9580cb47278f16f to start an election
I20240501 08:11:40.502571  8671 raft_consensus.cc:1067] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Signalling peer 0257c0e8038d4e3fa9580cb47278f16f to start an election
I20240501 08:11:40.503358  8007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b421e60e689c4adfabadb47bd9a9e167"
dest_uuid: "0257c0e8038d4e3fa9580cb47278f16f"
 from {username='slave'} at 127.2.248.129:57159
I20240501 08:11:40.503614  8007 raft_consensus.cc:482] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:40.503787  8007 raft_consensus.cc:3037] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:40.506739  8007 raft_consensus.cc:504] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:40.528585  6541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421e60e689c4adfabadb47bd9a9e167" candidate_uuid: "0257c0e8038d4e3fa9580cb47278f16f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
I20240501 08:11:40.528805  6541 raft_consensus.cc:3032] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:40.528900  6541 raft_consensus.cc:729] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:40.529186  6541 consensus_queue.cc:260] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:40.529551  6541 raft_consensus.cc:3037] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:40.537292  6541 raft_consensus.cc:2454] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0257c0e8038d4e3fa9580cb47278f16f in term 2.
I20240501 08:11:40.539047  7938 leader_election.cc:302] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0257c0e8038d4e3fa9580cb47278f16f, b0190f14862148f4bde78afbbf1ba43d; no voters: 
I20240501 08:11:40.545650  8007 leader_election.cc:288] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [CANDIDATE]: Term 2 election: Requested vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709), ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997)
I20240501 08:11:40.631975  8793 raft_consensus.cc:2781] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Leader election won for term 2
W20240501 08:11:40.632265  7938 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.248.129:41709, user_credentials={real_user=slave}} blocked reactor thread for 93295.3us
I20240501 08:11:40.642654  8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421e60e689c4adfabadb47bd9a9e167" candidate_uuid: "0257c0e8038d4e3fa9580cb47278f16f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ad679e52ba5643f2b11838d14696d1ed"
I20240501 08:11:40.643096  8274 raft_consensus.cc:3037] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:40.647221  8274 raft_consensus.cc:2454] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0257c0e8038d4e3fa9580cb47278f16f in term 2.
I20240501 08:11:40.655802  8793 raft_consensus.cc:686] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 LEADER]: Becoming Leader. State: Replica: 0257c0e8038d4e3fa9580cb47278f16f, State: Running, Role: LEADER
I20240501 08:11:40.656540  8793 consensus_queue.cc:237] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:40.675303  6398 catalog_manager.cc:5484] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f reported cstate change: term changed from 1 to 2, leader changed from b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) to 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133). New cstate: current_term: 2 leader_uuid: "0257c0e8038d4e3fa9580cb47278f16f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:40.785380  8758 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=106 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:40.788354  8758 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: Bootstrap complete.
I20240501 08:11:40.789180  8758 ts_tablet_manager.cc:1367] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: Time spent bootstrapping tablet: real 0.765s	user 0.156s	sys 0.014s
I20240501 08:11:40.792635  8758 raft_consensus.cc:348] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:40.794787  8758 raft_consensus.cc:729] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0257c0e8038d4e3fa9580cb47278f16f, State: Initialized, Role: LEARNER
I20240501 08:11:40.795639  8758 consensus_queue.cc:260] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 106, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: true } }
I20240501 08:11:40.796253  8758 ts_tablet_manager.cc:1398] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240501 08:11:40.801064  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session 0257c0e8038d4e3fa9580cb47278f16f-dff918bc67204080bf1c272c2d950045 received from {username='slave'} at 127.2.248.133:42855
I20240501 08:11:40.801218  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session 0257c0e8038d4e3fa9580cb47278f16f-dff918bc67204080bf1c272c2d950045 on tablet dff918bc67204080bf1c272c2d950045 with peer 0257c0e8038d4e3fa9580cb47278f16f
I20240501 08:11:40.950050  8007 raft_consensus.cc:1203] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.103->[1.104-1.120]   Dedup: 1.107->[1.108-1.120]
I20240501 08:11:40.991995  8797 raft_consensus.cc:1050] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER 0257c0e8038d4e3fa9580cb47278f16f to VOTER
I20240501 08:11:40.992666  8797 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } }
I20240501 08:11:40.996014  8004 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240501 08:11:40.996330  6800 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240501 08:11:40.998564  8671 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240501 08:11:40.998994  8671 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20240501 08:11:41.003238  8392 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index 104 to 122, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:41.005690  6800 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 104 to 122, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:41.008011  8006 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 104 to 122, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:41.013155  6398 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 104 to 122, 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:41.020496  6541 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } }
I20240501 08:11:41.035241  8007 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240501 08:11:41.037091  8392 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240501 08:11:41.046715  8392 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 122 to 123, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:41.050734  8007 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 122 to 123, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } }
I20240501 08:11:41.051017  6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dff918bc67204080bf1c272c2d950045 with cas_config_opid_index 122: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:41.057296  6397 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 122 to 123, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:41.061426  6781 tablet_service.cc:1514] Processing DeleteTablet for tablet dff918bc67204080bf1c272c2d950045 with delete_type TABLET_DATA_TOMBSTONED (TS 7fb652773c524f639dba8c189b7743ba not found in new config with opid_index 123) from {username='slave'} at 127.0.0.1:44490
I20240501 08:11:41.064452  8824 tablet_replica.cc:330] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: stopping tablet replica
I20240501 08:11:41.064745  8824 raft_consensus.cc:2229] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:41.065106  8824 raft_consensus.cc:2258] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:41.066169  8824 ts_tablet_manager.cc:1875] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:41.076346  8824 ts_tablet_manager.cc:1888] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.122
I20240501 08:11:41.076577  8824 log.cc:1196] T dff918bc67204080bf1c272c2d950045 P 7fb652773c524f639dba8c189b7743ba: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/dff918bc67204080bf1c272c2d950045
I20240501 08:11:41.077920  6386 catalog_manager.cc:4830] TS 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453): tablet dff918bc67204080bf1c272c2d950045 (table rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]) successfully deleted
I20240501 08:11:41.152293  6541 raft_consensus.cc:1261] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Refusing update from remote peer 0257c0e8038d4e3fa9580cb47278f16f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:11:41.153281  8793 consensus_queue.cc:1035] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:41.196075  8275 raft_consensus.cc:1261] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Refusing update from remote peer 0257c0e8038d4e3fa9580cb47278f16f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:11:41.196830  8793 consensus_queue.cc:1035] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:41.810966  8007 consensus_queue.cc:237] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:41.815629  8274 raft_consensus.cc:1261] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Refusing update from remote peer 0257c0e8038d4e3fa9580cb47278f16f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:41.816260  8833 consensus_queue.cc:1035] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:41.828910  8833 raft_consensus.cc:2932] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:41.832355  6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b421e60e689c4adfabadb47bd9a9e167 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:41.833122  8271 raft_consensus.cc:2932] T b421e60e689c4adfabadb47bd9a9e167 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:41.834872  6398 catalog_manager.cc:5484] T b421e60e689c4adfabadb47bd9a9e167 P 0257c0e8038d4e3fa9580cb47278f16f reported cstate change: config changed from index 3 to 5, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "0257c0e8038d4e3fa9580cb47278f16f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:41.843200  6522 tablet_service.cc:1514] Processing DeleteTablet for tablet b421e60e689c4adfabadb47bd9a9e167 with delete_type TABLET_DATA_TOMBSTONED (TS b0190f14862148f4bde78afbbf1ba43d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51260
I20240501 08:11:41.881556  8892 tablet_replica.cc:330] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: stopping tablet replica
I20240501 08:11:41.882262  8892 raft_consensus.cc:2229] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:41.887603  8892 raft_consensus.cc:2258] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:41.896519  8892 ts_tablet_manager.cc:1875] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:41.935356  8892 ts_tablet_manager.cc:1888] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:11:41.937072  8892 log.cc:1196] T b421e60e689c4adfabadb47bd9a9e167 P b0190f14862148f4bde78afbbf1ba43d: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/b421e60e689c4adfabadb47bd9a9e167
I20240501 08:11:41.940812  6385 catalog_manager.cc:4830] TS b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709): tablet b421e60e689c4adfabadb47bd9a9e167 (table TestTable [id=81f06f1786d74ce48c8a6680e52f03ae]) successfully deleted
I20240501 08:11:42.541899  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:42.577255  7986 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:42.648276  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:42.672395  6652 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:42.685736  6522 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:42.686081  8118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:42.823763  7854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:43.024453  6670 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 1.145, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:43.026973  7874 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240501 08:11:43.030781  8811 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 144, Time since last communication: 0.000s
W20240501 08:11:43.032565  6608 consensus_peers.cc:485] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 -> Peer c284970db26b4c95a9ee09310b5adcbe (127.2.248.134:33609): Couldn't send request to peer c284970db26b4c95a9ee09310b5adcbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 217060ad409b4492a7d4ce043578aa49. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:43.035462  8811 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.146: config changed from index 75 to 146, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:43.037724  7871 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 75 to 146, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } } }
I20240501 08:11:43.048241  6394 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 75 to 146, NON_VOTER c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:43.190829  6541 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 1.146, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:43.199295  8007 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch)
I20240501 08:11:43.200698  8797 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I20240501 08:11:43.204752  8371 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.147: config changed from index 84 to 147, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:43.208155  8005 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index 84 to 147, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
I20240501 08:11:43.213981  6394 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f reported cstate change: config changed from index 84 to 147, NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } } }
W20240501 08:11:43.229799  6477 consensus_peers.cc:485] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d -> Peer fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129): Couldn't send request to peer fcfccbafc3254a87896f0e72c8078ab5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527072daf1ea4b478a1b35f4a5c4bf3c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:43.318380  6541 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.174, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
W20240501 08:11:43.324256  6478 consensus_peers.cc:485] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d -> Peer ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997): Couldn't send request to peer ad679e52ba5643f2b11838d14696d1ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dff918bc67204080bf1c272c2d950045. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:43.330121  8006 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240501 08:11:43.333808  8371 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240501 08:11:43.339357  8371 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.175: config changed from index 123 to 175, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:43.339958  8006 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 123 to 175, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:43.344986  6397 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f reported cstate change: config changed from index 123 to 175, NON_VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } } }
I20240501 08:11:43.546082  8990 ts_tablet_manager.cc:902] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: Initiating tablet copy from peer d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:43.573223  8990 tablet_copy_client.cc:308] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45487
I20240501 08:11:43.573963  6692 tablet_copy_service.cc:140] P d143a9ab17334a6081de63802bdce654: Received BeginTabletCopySession request for tablet 217060ad409b4492a7d4ce043578aa49 from peer c284970db26b4c95a9ee09310b5adcbe ({username='slave'} at 127.2.248.134:47593)
I20240501 08:11:43.574116  6692 tablet_copy_service.cc:161] P d143a9ab17334a6081de63802bdce654: Beginning new tablet copy session on tablet 217060ad409b4492a7d4ce043578aa49 from peer c284970db26b4c95a9ee09310b5adcbe at {username='slave'} at 127.2.248.134:47593: session id = c284970db26b4c95a9ee09310b5adcbe-217060ad409b4492a7d4ce043578aa49
I20240501 08:11:43.576895  6692 tablet_copy_source_session.cc:205] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:43.578583  8990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217060ad409b4492a7d4ce043578aa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.594262  8990 tablet_copy_client.cc:783] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:43.594643  8990 tablet_copy_client.cc:647] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:43.600791  8990 tablet_copy_client.cc:523] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:43.607753  8990 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap starting.
I20240501 08:11:43.812541  8998 ts_tablet_manager.cc:902] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:43.822676  9000 ts_tablet_manager.cc:902] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:44.172345  9000 tablet_copy_client.cc:308] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:44.173624  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet dff918bc67204080bf1c272c2d950045 from peer ad679e52ba5643f2b11838d14696d1ed ({username='slave'} at 127.2.248.135:55761)
I20240501 08:11:44.173792  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet dff918bc67204080bf1c272c2d950045 from peer ad679e52ba5643f2b11838d14696d1ed at {username='slave'} at 127.2.248.135:55761: session id = ad679e52ba5643f2b11838d14696d1ed-dff918bc67204080bf1c272c2d950045
I20240501 08:11:44.176906  6562 tablet_copy_source_session.cc:205] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:44.178501  9000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff918bc67204080bf1c272c2d950045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.189798  8998 tablet_copy_client.cc:308] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:44.200018  9000 tablet_copy_client.cc:783] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:44.200302  9000 tablet_copy_client.cc:647] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:44.209630  9000 tablet_copy_client.cc:523] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:44.215535  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet 527072daf1ea4b478a1b35f4a5c4bf3c from peer fcfccbafc3254a87896f0e72c8078ab5 ({username='slave'} at 127.2.248.132:44645)
I20240501 08:11:44.215731  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet 527072daf1ea4b478a1b35f4a5c4bf3c from peer fcfccbafc3254a87896f0e72c8078ab5 at {username='slave'} at 127.2.248.132:44645: session id = fcfccbafc3254a87896f0e72c8078ab5-527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:11:44.218771  6562 tablet_copy_source_session.cc:205] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:44.221026  8998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527072daf1ea4b478a1b35f4a5c4bf3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.221335  9000 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap starting.
I20240501 08:11:44.227757  8998 tablet_copy_client.cc:783] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:44.228050  8998 tablet_copy_client.cc:647] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:44.231809  8998 tablet_copy_client.cc:523] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:44.236408  8998 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap starting.
I20240501 08:11:44.298630  8990 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=157 overwritten=0 applied=157 ignored=0} inserts{seen=839 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:44.299324  8990 tablet_bootstrap.cc:492] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: Bootstrap complete.
I20240501 08:11:44.299783  8990 ts_tablet_manager.cc:1367] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: Time spent bootstrapping tablet: real 0.692s	user 0.228s	sys 0.020s
I20240501 08:11:44.300909  8990 raft_consensus.cc:348] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:44.302028  8990 raft_consensus.cc:729] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c284970db26b4c95a9ee09310b5adcbe, State: Initialized, Role: LEARNER
I20240501 08:11:44.302285  8990 consensus_queue.cc:260] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 157, Last appended: 1.157, Last appended by leader: 157, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: true } }
I20240501 08:11:44.309566  8990 ts_tablet_manager.cc:1398] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20240501 08:11:44.314294  6692 tablet_copy_service.cc:342] P d143a9ab17334a6081de63802bdce654: Request end of tablet copy session c284970db26b4c95a9ee09310b5adcbe-217060ad409b4492a7d4ce043578aa49 received from {username='slave'} at 127.2.248.134:47593
I20240501 08:11:44.314450  6692 tablet_copy_service.cc:433] P d143a9ab17334a6081de63802bdce654: ending tablet copy session c284970db26b4c95a9ee09310b5adcbe-217060ad409b4492a7d4ce043578aa49 on tablet 217060ad409b4492a7d4ce043578aa49 with peer c284970db26b4c95a9ee09310b5adcbe
I20240501 08:11:44.533720  8138 raft_consensus.cc:1203] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.145->[1.146-1.175]   Dedup: 1.157->[1.158-1.175]
I20240501 08:11:44.546160  8998 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=161 overwritten=0 applied=160 ignored=0} inserts{seen=842 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:44.546940  8998 tablet_bootstrap.cc:492] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: Bootstrap complete.
I20240501 08:11:44.547389  8998 ts_tablet_manager.cc:1367] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: Time spent bootstrapping tablet: real 0.311s	user 0.223s	sys 0.014s
I20240501 08:11:44.548467  8998 raft_consensus.cc:348] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:44.549314  8998 raft_consensus.cc:729] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fcfccbafc3254a87896f0e72c8078ab5, State: Initialized, Role: LEARNER
I20240501 08:11:44.549615  8998 consensus_queue.cc:260] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 1.161, Last appended by leader: 161, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: true } }
I20240501 08:11:44.563306  8998 ts_tablet_manager.cc:1398] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.007s
I20240501 08:11:44.564392  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session fcfccbafc3254a87896f0e72c8078ab5-527072daf1ea4b478a1b35f4a5c4bf3c received from {username='slave'} at 127.2.248.132:44645
I20240501 08:11:44.564528  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session fcfccbafc3254a87896f0e72c8078ab5-527072daf1ea4b478a1b35f4a5c4bf3c on tablet 527072daf1ea4b478a1b35f4a5c4bf3c with peer fcfccbafc3254a87896f0e72c8078ab5
I20240501 08:11:44.593130  8498 raft_consensus.cc:1050] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: attempting to promote NON_VOTER c284970db26b4c95a9ee09310b5adcbe to VOTER
I20240501 08:11:44.593922  8498 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 176, Committed index: 176, Last appended: 1.176, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:44.595963  8137 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 1 LEARNER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240501 08:11:44.596524  7872 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20240501 08:11:44.597206  8498 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20240501 08:11:44.597425  8808 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20240501 08:11:44.599846  8808 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.177: config changed from index 146 to 177, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:44.603830  8137 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 146 to 177, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:44.604044  7872 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 146 to 177, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:44.609977  6394 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 reported cstate change: config changed from index 146 to 177, c284970db26b4c95a9ee09310b5adcbe (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:44.618424  9000 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=202 overwritten=0 applied=201 ignored=0} inserts{seen=829 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:44.619426  9000 tablet_bootstrap.cc:492] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: Bootstrap complete.
I20240501 08:11:44.619741  9000 ts_tablet_manager.cc:1367] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: Time spent bootstrapping tablet: real 0.399s	user 0.265s	sys 0.041s
I20240501 08:11:44.620510  9000 raft_consensus.cc:348] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:44.621940  9000 raft_consensus.cc:729] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad679e52ba5643f2b11838d14696d1ed, State: Initialized, Role: LEARNER
I20240501 08:11:44.622423  9000 consensus_queue.cc:260] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 201, Last appended: 1.202, Last appended by leader: 202, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: true } }
I20240501 08:11:44.774473  9000 ts_tablet_manager.cc:1398] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed: Time spent starting tablet: real 0.155s	user 0.047s	sys 0.059s
I20240501 08:11:44.775303  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session ad679e52ba5643f2b11838d14696d1ed-dff918bc67204080bf1c272c2d950045 received from {username='slave'} at 127.2.248.135:55761
I20240501 08:11:44.775451  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session ad679e52ba5643f2b11838d14696d1ed-dff918bc67204080bf1c272c2d950045 on tablet dff918bc67204080bf1c272c2d950045 with peer ad679e52ba5643f2b11838d14696d1ed
I20240501 08:11:44.852133  7871 raft_consensus.cc:1203] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.146->[1.147-1.167]   Dedup: 1.161->[1.162-1.167]
I20240501 08:11:44.902833  8274 raft_consensus.cc:1203] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.174->[1.175-1.210]   Dedup: 1.202->[1.203-1.210]
I20240501 08:11:44.983079  8392 raft_consensus.cc:1050] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER fcfccbafc3254a87896f0e72c8078ab5 to VOTER
I20240501 08:11:44.983738  8392 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.168, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:44.986826  7871 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240501 08:11:44.990388  8371 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240501 08:11:44.990531  9048 raft_consensus.cc:1050] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER ad679e52ba5643f2b11838d14696d1ed to VOTER
I20240501 08:11:44.991533  9048 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.212, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:44.993899  8274 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240501 08:11:44.994251  8006 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 211. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240501 08:11:44.994611  9001 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240501 08:11:44.996609  8797 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240501 08:11:44.997448  8392 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.213: config changed from index 175 to 213, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:44.998234  8004 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240501 08:11:45.003531  8275 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 175 to 213, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:45.004730  9001 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 167, Time since last communication: 0.000s
I20240501 08:11:45.007143  8005 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 175 to 213, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:45.009090  8797 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.169: config changed from index 147 to 169, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:45.012387  6398 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed reported cstate change: config changed from index 175 to 213, ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:45.019637  7871 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 147 to 169, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:45.037904  6394 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 reported cstate change: config changed from index 147 to 169, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:45.059062  8006 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 147 to 169, fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:45.135255  6670 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "217060ad409b4492a7d4ce043578aa49"
dest_uuid: "d143a9ab17334a6081de63802bdce654"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46922
I20240501 08:11:45.135540  6670 raft_consensus.cc:595] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:11:45.140890  6630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.161060  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.198230  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.205734  6630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.243688  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.259038  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.282410  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.320290  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.338230  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.353421  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.370420  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.384411  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.400195  6630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.415668  6630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
W20240501 08:11:45.435531  6630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
I20240501 08:11:45.440757  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dff918bc67204080bf1c272c2d950045"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39454
I20240501 08:11:45.440994  6541 raft_consensus.cc:595] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
W20240501 08:11:45.456899  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
I20240501 08:11:45.462388  9055 raft_consensus.cc:979] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: : Instructing follower c284970db26b4c95a9ee09310b5adcbe to start an election
I20240501 08:11:45.462592  9055 raft_consensus.cc:1067] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Signalling peer c284970db26b4c95a9ee09310b5adcbe to start an election
I20240501 08:11:45.466114  8138 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "217060ad409b4492a7d4ce043578aa49"
dest_uuid: "c284970db26b4c95a9ee09310b5adcbe"
 from {username='slave'} at 127.2.248.130:60569
I20240501 08:11:45.466645  8138 raft_consensus.cc:482] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:45.466935  8138 raft_consensus.cc:3037] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:45.482787  8138 raft_consensus.cc:504] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
W20240501 08:11:45.484047  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Service unavailable: leader transfer in progress
I20240501 08:11:45.491358  6670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217060ad409b4492a7d4ce043578aa49" candidate_uuid: "c284970db26b4c95a9ee09310b5adcbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 184 } } ignore_live_leader: true dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:45.491662  6670 raft_consensus.cc:3032] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:45.491794  6670 raft_consensus.cc:729] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Running, Role: LEADER
I20240501 08:11:45.492265  6670 consensus_queue.cc:260] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 1.184, Last appended by leader: 184, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:45.492861  6670 raft_consensus.cc:3037] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:45.497125  6670 raft_consensus.cc:2454] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c284970db26b4c95a9ee09310b5adcbe in term 2.
I20240501 08:11:45.497889  8072 leader_election.cc:302] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c284970db26b4c95a9ee09310b5adcbe, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:45.505702  9092 raft_consensus.cc:2781] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 2 FOLLOWER]: Leader election won for term 2
W20240501 08:11:45.505692  6631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46252: Illegal state: replica d143a9ab17334a6081de63802bdce654 is not leader of this config: current role FOLLOWER
I20240501 08:11:45.513139  9092 raft_consensus.cc:686] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 2 LEADER]: Becoming Leader. State: Replica: c284970db26b4c95a9ee09310b5adcbe, State: Running, Role: LEADER
I20240501 08:11:45.513804  9092 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 1.184, Last appended by leader: 184, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:45.517261  8138 leader_election.cc:288] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [CANDIDATE]: Term 2 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487), fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129)
I20240501 08:11:45.533180  6394 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: term changed from 1 to 2, leader changed from d143a9ab17334a6081de63802bdce654 (127.2.248.130) to c284970db26b4c95a9ee09310b5adcbe (127.2.248.134). New cstate: current_term: 2 leader_uuid: "c284970db26b4c95a9ee09310b5adcbe" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:45.542819  7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217060ad409b4492a7d4ce043578aa49" candidate_uuid: "c284970db26b4c95a9ee09310b5adcbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 184 } } ignore_live_leader: true dest_uuid: "fcfccbafc3254a87896f0e72c8078ab5"
I20240501 08:11:45.543148  7871 raft_consensus.cc:3037] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:45.546046  7871 raft_consensus.cc:2454] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c284970db26b4c95a9ee09310b5adcbe in term 2.
I20240501 08:11:45.614570  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "527072daf1ea4b478a1b35f4a5c4bf3c"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39470
I20240501 08:11:45.614809  6541 raft_consensus.cc:595] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
W20240501 08:11:45.620354  6500 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.628086  6500 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.628298  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.635190  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.638446  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.643841  6500 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.643842  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.651790  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.653077  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:45.656929  7872 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 2 FOLLOWER]: Refusing update from remote peer c284970db26b4c95a9ee09310b5adcbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 184. Preceding OpId from leader: term: 2 index: 186. (index mismatch)
I20240501 08:11:45.657863  9093 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I20240501 08:11:45.666308  6670 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Refusing update from remote peer c284970db26b4c95a9ee09310b5adcbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 184. Preceding OpId from leader: term: 2 index: 186. (index mismatch)
I20240501 08:11:45.666975  9093 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I20240501 08:11:45.670960  9031 mvcc.cc:205] Tried to move back new op lower bound from 7022801328762904576 to 7022801328199598080. Current Snapshot: MvccSnapshot[applied={T|T < 7022801328762904576 or (T in {7022801328762904576})}]
W20240501 08:11:45.676535  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.680383  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:45.688305  9009 mvcc.cc:205] Tried to move back new op lower bound from 7022801328762904576 to 7022801328199598080. Current Snapshot: MvccSnapshot[applied={T|T < 7022801328762904576}]
W20240501 08:11:45.692787  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.696050  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.704558  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.708170  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.720261  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.723004  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
I20240501 08:11:45.728423  8136 consensus_queue.cc:237] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 2.187, Last appended by leader: 184, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } }
I20240501 08:11:45.730890  7874 raft_consensus.cc:1261] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 2 FOLLOWER]: Refusing update from remote peer c284970db26b4c95a9ee09310b5adcbe: Log matching property violated. Preceding OpId in replica: term: 2 index: 187. Preceding OpId from leader: term: 2 index: 188. (index mismatch)
W20240501 08:11:45.732213  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:45.732209  9092 consensus_queue.cc:1035] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
W20240501 08:11:45.735147  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
I20240501 08:11:45.735421  9093 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe [term 2 LEADER]: Committing config change with OpId 2.188: config changed from index 177 to 188, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:45.740456  6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 217060ad409b4492a7d4ce043578aa49 with cas_config_opid_index 177: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:45.742580  7874 raft_consensus.cc:2932] T 217060ad409b4492a7d4ce043578aa49 P fcfccbafc3254a87896f0e72c8078ab5 [term 2 FOLLOWER]: Committing config change with OpId 2.188: config changed from index 177 to 188, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } }
I20240501 08:11:45.742887  6397 catalog_manager.cc:5484] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: config changed from index 177 to 188, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New cstate: current_term: 2 leader_uuid: "c284970db26b4c95a9ee09310b5adcbe" committed_config { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:11:45.744285  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:45.749424  6652 tablet_service.cc:1514] Processing DeleteTablet for tablet 217060ad409b4492a7d4ce043578aa49 with delete_type TABLET_DATA_TOMBSTONED (TS d143a9ab17334a6081de63802bdce654 not found in new config with opid_index 188) from {username='slave'} at 127.0.0.1:54172
W20240501 08:11:45.751734  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.759732  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.765326  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.774564  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:45.776427  9116 tablet_replica.cc:330] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: stopping tablet replica
I20240501 08:11:45.776813  9116 raft_consensus.cc:2229] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:45.777171  9116 raft_consensus.cc:2258] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:45.779167  9116 ts_tablet_manager.cc:1875] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:11:45.781805  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
I20240501 08:11:45.786322  9116 ts_tablet_manager.cc:1888] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.187
I20240501 08:11:45.786456  9116 log.cc:1196] T 217060ad409b4492a7d4ce043578aa49 P d143a9ab17334a6081de63802bdce654: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/217060ad409b4492a7d4ce043578aa49
I20240501 08:11:45.787909  6386 catalog_manager.cc:4830] TS d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487): tablet 217060ad409b4492a7d4ce043578aa49 (table rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]) successfully deleted
W20240501 08:11:45.792953  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.795622  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.812572  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.814056  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.832373  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.832373  6500 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.852840  6500 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.856951  6500 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.878840  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
W20240501 08:11:45.880457  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:45.903004  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dff918bc67204080bf1c272c2d950045"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39490
I20240501 08:11:45.903198  6541 raft_consensus.cc:595] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
W20240501 08:11:45.910903  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.917112  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
I20240501 08:11:45.946292  9071 raft_consensus.cc:979] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: : Instructing follower 0257c0e8038d4e3fa9580cb47278f16f to start an election
I20240501 08:11:45.946534  9071 raft_consensus.cc:1067] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Signalling peer 0257c0e8038d4e3fa9580cb47278f16f to start an election
W20240501 08:11:45.947415  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:45.948544  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Service unavailable: leader transfer in progress
I20240501 08:11:45.951476  8004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dff918bc67204080bf1c272c2d950045"
dest_uuid: "0257c0e8038d4e3fa9580cb47278f16f"
 from {username='slave'} at 127.2.248.129:57159
I20240501 08:11:45.951699  8004 raft_consensus.cc:482] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:45.951844  8004 raft_consensus.cc:3037] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:45.954594  8004 raft_consensus.cc:504] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:45.959048  6541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff918bc67204080bf1c272c2d950045" candidate_uuid: "0257c0e8038d4e3fa9580cb47278f16f" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: true dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
I20240501 08:11:45.959240  6541 raft_consensus.cc:3032] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:45.959327  6541 raft_consensus.cc:729] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:45.959560  6541 consensus_queue.cc:260] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 1.224, Last appended by leader: 224, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:45.959863  6541 raft_consensus.cc:3037] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:45.960829  8271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff918bc67204080bf1c272c2d950045" candidate_uuid: "0257c0e8038d4e3fa9580cb47278f16f" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: true dest_uuid: "ad679e52ba5643f2b11838d14696d1ed"
I20240501 08:11:45.961019  8271 raft_consensus.cc:3037] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:45.972690  6541 raft_consensus.cc:2454] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0257c0e8038d4e3fa9580cb47278f16f in term 2.
I20240501 08:11:45.974779  8271 raft_consensus.cc:2454] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0257c0e8038d4e3fa9580cb47278f16f in term 2.
I20240501 08:11:45.975409  7941 leader_election.cc:302] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0257c0e8038d4e3fa9580cb47278f16f, ad679e52ba5643f2b11838d14696d1ed; no voters: 
I20240501 08:11:45.975786  9028 raft_consensus.cc:2781] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:45.978334  8004 leader_election.cc:288] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [CANDIDATE]: Term 2 election: Requested vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709), ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997)
W20240501 08:11:45.974241  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47860: Illegal state: replica b0190f14862148f4bde78afbbf1ba43d is not leader of this config: current role FOLLOWER
W20240501 08:11:45.977099  6502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:45.991544  9028 raft_consensus.cc:686] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 LEADER]: Becoming Leader. State: Replica: 0257c0e8038d4e3fa9580cb47278f16f, State: Running, Role: LEADER
I20240501 08:11:45.992020  9028 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 1.224, Last appended by leader: 224, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:46.001544  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "527072daf1ea4b478a1b35f4a5c4bf3c"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39518
I20240501 08:11:46.001711  6541 raft_consensus.cc:595] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:45.996994  6397 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f reported cstate change: term changed from 1 to 2, leader changed from b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) to 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133). New cstate: current_term: 2 leader_uuid: "0257c0e8038d4e3fa9580cb47278f16f" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:46.011969  6501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:46.037377  6502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:46.065768  6502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:46.075837  9071 raft_consensus.cc:979] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: : Instructing follower fcfccbafc3254a87896f0e72c8078ab5 to start an election
I20240501 08:11:46.076454  9071 raft_consensus.cc:1067] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Signalling peer fcfccbafc3254a87896f0e72c8078ab5 to start an election
I20240501 08:11:46.078336  7872 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "527072daf1ea4b478a1b35f4a5c4bf3c"
dest_uuid: "fcfccbafc3254a87896f0e72c8078ab5"
 from {username='slave'} at 127.2.248.129:37635
I20240501 08:11:46.079324  7872 raft_consensus.cc:482] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:46.079577  7872 raft_consensus.cc:3037] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:46.082707  7872 raft_consensus.cc:504] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
W20240501 08:11:46.095773  6502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240501 08:11:46.125779  6502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240501 08:11:46.130729  6541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527072daf1ea4b478a1b35f4a5c4bf3c" candidate_uuid: "fcfccbafc3254a87896f0e72c8078ab5" candidate_term: 2 candidate_status { last_received { term: 1 index: 191 } } ignore_live_leader: true dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
I20240501 08:11:46.130910  6541 raft_consensus.cc:3032] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:46.131013  6541 raft_consensus.cc:729] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:46.131312  6541 consensus_queue.cc:260] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 1.191, Last appended by leader: 191, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:46.131671  6541 raft_consensus.cc:3037] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:46.134212  6541 raft_consensus.cc:2454] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfccbafc3254a87896f0e72c8078ab5 in term 2.
I20240501 08:11:46.138751  7807 leader_election.cc:302] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, fcfccbafc3254a87896f0e72c8078ab5; no voters: 
I20240501 08:11:46.140527  9111 raft_consensus.cc:2781] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:46.145876  7872 leader_election.cc:288] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [CANDIDATE]: Term 2 election: Requested vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709), 0257c0e8038d4e3fa9580cb47278f16f (127.2.248.133:43177)
W20240501 08:11:46.157912  6502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Illegal state: replica b0190f14862148f4bde78afbbf1ba43d is not leader of this config: current role FOLLOWER
I20240501 08:11:46.160228  9111 raft_consensus.cc:686] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 2 LEADER]: Becoming Leader. State: Replica: fcfccbafc3254a87896f0e72c8078ab5, State: Running, Role: LEADER
I20240501 08:11:46.160818  9111 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 1.191, Last appended by leader: 191, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:46.170821  6397 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 reported cstate change: term changed from 1 to 2, leader changed from b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) to fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132). New cstate: current_term: 2 leader_uuid: "fcfccbafc3254a87896f0e72c8078ab5" committed_config { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:46.202551  8004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527072daf1ea4b478a1b35f4a5c4bf3c" candidate_uuid: "fcfccbafc3254a87896f0e72c8078ab5" candidate_term: 2 candidate_status { last_received { term: 1 index: 191 } } ignore_live_leader: true dest_uuid: "0257c0e8038d4e3fa9580cb47278f16f"
I20240501 08:11:46.202981  8004 raft_consensus.cc:3037] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:46.206337  8275 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Refusing update from remote peer 0257c0e8038d4e3fa9580cb47278f16f: Log matching property violated. Preceding OpId in replica: term: 1 index: 224. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20240501 08:11:46.208065  9137 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20240501 08:11:46.209352  6541 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Refusing update from remote peer 0257c0e8038d4e3fa9580cb47278f16f: Log matching property violated. Preceding OpId in replica: term: 1 index: 224. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20240501 08:11:46.210160  9137 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20240501 08:11:46.213461  8004 raft_consensus.cc:2454] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfccbafc3254a87896f0e72c8078ab5 in term 2.
I20240501 08:11:46.223219  9035 mvcc.cc:205] Tried to move back new op lower bound from 7022801331005026304 to 7022801330146967552. Current Snapshot: MvccSnapshot[applied={T|T < 7022801331005026304 or (T in {7022801331005026304})}]
I20240501 08:11:46.250202  8004 consensus_queue.cc:237] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 224, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } }
I20240501 08:11:46.251653  8271 raft_consensus.cc:1261] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Refusing update from remote peer 0257c0e8038d4e3fa9580cb47278f16f: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20240501 08:11:46.253897  9137 consensus_queue.cc:1035] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20240501 08:11:46.256330  9137 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P 0257c0e8038d4e3fa9580cb47278f16f [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 213 to 228, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:46.260035  8271 raft_consensus.cc:2932] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 213 to 228, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:46.269402  6397 catalog_manager.cc:5484] T dff918bc67204080bf1c272c2d950045 P ad679e52ba5643f2b11838d14696d1ed reported cstate change: config changed from index 213 to 228, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "0257c0e8038d4e3fa9580cb47278f16f" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } }
I20240501 08:11:46.275455  6522 tablet_service.cc:1514] Processing DeleteTablet for tablet dff918bc67204080bf1c272c2d950045 with delete_type TABLET_DATA_TOMBSTONED (TS b0190f14862148f4bde78afbbf1ba43d not found in new config with opid_index 228) from {username='slave'} at 127.0.0.1:51260
I20240501 08:11:46.283411  6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dff918bc67204080bf1c272c2d950045 with cas_config_opid_index 213: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:46.296005  9153 tablet_replica.cc:330] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: stopping tablet replica
I20240501 08:11:46.297266  9153 raft_consensus.cc:2229] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:46.297540  9153 pending_rounds.cc:62] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Trying to abort 1 pending ops.
I20240501 08:11:46.297686  9153 pending_rounds.cc:68] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Aborting op as it isn't in flight: id { term: 2 index: 227 } timestamp: 7022801331165708288 op_type: WRITE_OP write_request { tablet_id: "dff918bc67204080bf1c272c2d950045" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000M\245\177{m\266:=\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""e4631fe2cf84c3d6" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022801331005026304 authz_token { token_data: "\010\342\364\307\261\006\"3\n\005slave\022*\n 42721b580ee34e96ba5ba4d773108702\020\001\030\001 \001(\001" signature: "<redacted>""\206\212\201\367\302\313v\200\273HCC\356\237\377i\207\272\211*I\371\004.\237x\3625\250\273\r\326R\310\265\244\317\021c) AE8\273 \t\302^\213\256\221\017\3664q\017K\272\301 \261\336\'" signing_key_seq_num: 0 } } request_id { client_id: "66c43c95845c4cbab7d8c8bb28a0ab99" seq_no: 596 first_incomplete_seq_no: 596 attempt_no: 0 }
I20240501 08:11:46.314901  9153 raft_consensus.cc:2258] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:46.337924  6541 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Refusing update from remote peer fcfccbafc3254a87896f0e72c8078ab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 2 index: 193. (index mismatch)
I20240501 08:11:46.338699  9142 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I20240501 08:11:46.341156  8004 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Refusing update from remote peer fcfccbafc3254a87896f0e72c8078ab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 2 index: 193. (index mismatch)
I20240501 08:11:46.341723  9142 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I20240501 08:11:46.380587  9153 ts_tablet_manager.cc:1875] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:46.408540  9153 ts_tablet_manager.cc:1888] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.227
I20240501 08:11:46.408779  9153 log.cc:1196] T dff918bc67204080bf1c272c2d950045 P b0190f14862148f4bde78afbbf1ba43d: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/dff918bc67204080bf1c272c2d950045
I20240501 08:11:46.410284  6385 catalog_manager.cc:4830] TS b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709): tablet dff918bc67204080bf1c272c2d950045 (table rebalance_test_table_1 [id=42721b580ee34e96ba5ba4d773108702]) successfully deleted
I20240501 08:11:46.494141  7872 consensus_queue.cc:237] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 2.195, Last appended by leader: 191, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } }
I20240501 08:11:46.510108  8004 raft_consensus.cc:1261] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Refusing update from remote peer fcfccbafc3254a87896f0e72c8078ab5: Log matching property violated. Preceding OpId in replica: term: 2 index: 195. Preceding OpId from leader: term: 2 index: 196. (index mismatch)
I20240501 08:11:46.511567  9142 consensus_queue.cc:1035] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I20240501 08:11:46.518327  9111 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P fcfccbafc3254a87896f0e72c8078ab5 [term 2 LEADER]: Committing config change with OpId 2.196: config changed from index 169 to 196, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:46.531999  8007 raft_consensus.cc:2932] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f [term 2 FOLLOWER]: Committing config change with OpId 2.196: config changed from index 169 to 196, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:46.539247  6397 catalog_manager.cc:5484] T 527072daf1ea4b478a1b35f4a5c4bf3c P 0257c0e8038d4e3fa9580cb47278f16f reported cstate change: config changed from index 169 to 196, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "fcfccbafc3254a87896f0e72c8078ab5" committed_config { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "0257c0e8038d4e3fa9580cb47278f16f" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43177 } attrs { promote: false } } peers { permanent_uuid: "fcfccbafc3254a87896f0e72c8078ab5" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 36129 } attrs { promote: false } } }
I20240501 08:11:46.547405  6387 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 527072daf1ea4b478a1b35f4a5c4bf3c with cas_config_opid_index 169: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:46.549389  6522 tablet_service.cc:1514] Processing DeleteTablet for tablet 527072daf1ea4b478a1b35f4a5c4bf3c with delete_type TABLET_DATA_TOMBSTONED (TS b0190f14862148f4bde78afbbf1ba43d not found in new config with opid_index 196) from {username='slave'} at 127.0.0.1:51260
I20240501 08:11:46.555222  9153 tablet_replica.cc:330] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: stopping tablet replica
I20240501 08:11:46.556293  9153 raft_consensus.cc:2229] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:46.556560  9153 raft_consensus.cc:2258] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:46.564937  9153 ts_tablet_manager.cc:1875] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:46.601207  9153 ts_tablet_manager.cc:1888] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.195
I20240501 08:11:46.601433  9153 log.cc:1196] T 527072daf1ea4b478a1b35f4a5c4bf3c P b0190f14862148f4bde78afbbf1ba43d: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:11:46.602350  6385 catalog_manager.cc:4830] TS b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709): tablet 527072daf1ea4b478a1b35f4a5c4bf3c (table rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]) successfully deleted
I20240501 08:11:47.274542  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.283629  6522 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:47.320793  8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:47.333623  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:47.447258  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:47.551421  6652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:47.558712  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:48.108615  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:48.142503  6522 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:48.182068  8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:48.256614  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.289853  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:48.530421  6652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:48.545531  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:11:49.078014  6476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.130:45487, user_credentials={real_user=slave}} blocked reactor thread for 50694us
I20240501 08:11:50.705439  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:50.713511  6652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:50.743345  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:50.768190  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:50.769932  6522 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:50.785399  8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:50.898326  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:11:50.915009  6606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.131:43453, user_credentials={real_user=slave}} blocked reactor thread for 59205us
I20240501 08:11:51.588840  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:51.713819  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:51.724520  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:51.765370  6522 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:51.781548  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:51.781590  8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:51.783143  6652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:52.728471  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:52.745994  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:52.785972  6652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:52.795493  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:52.798072  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:52.804956  8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:52.836042  6522 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 42d8139962d445eba456f56990374af2 | 127.2.248.190:39497 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:42939 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 0257c0e8038d4e3fa9580cb47278f16f | 127.2.248.133:43177 | HEALTHY | <none>   |       2        |       0
 7fb652773c524f639dba8c189b7743ba | 127.2.248.131:43453 | HEALTHY | <none>   |       3        |       0
 ad679e52ba5643f2b11838d14696d1ed | 127.2.248.135:36997 | HEALTHY | <none>   |       0        |       0
 b0190f14862148f4bde78afbbf1ba43d | 127.2.248.129:41709 | HEALTHY | <none>   |       5        |       0
 c284970db26b4c95a9ee09310b5adcbe | 127.2.248.134:33609 | HEALTHY | <none>   |       1        |       0
 d143a9ab17334a6081de63802bdce654 | 127.2.248.130:45487 | HEALTHY | <none>   |       4        |       0
 fcfccbafc3254a87896f0e72c8078ab5 | 127.2.248.132:36129 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.2.248.129                                                                                                                                                   | experimental          | 127.2.248.129:41709
 local_ip_for_outbound_sockets            | 127.2.248.130                                                                                                                                                   | experimental          | 127.2.248.130:45487
 local_ip_for_outbound_sockets            | 127.2.248.131                                                                                                                                                   | experimental          | 127.2.248.131:43453
 local_ip_for_outbound_sockets            | 127.2.248.132                                                                                                                                                   | experimental          | 127.2.248.132:36129
 local_ip_for_outbound_sockets            | 127.2.248.133                                                                                                                                                   | experimental          | 127.2.248.133:43177
 local_ip_for_outbound_sockets            | 127.2.248.134                                                                                                                                                   | experimental          | 127.2.248.134:33609
 local_ip_for_outbound_sockets            | 127.2.248.135                                                                                                                                                   | experimental          | 127.2.248.135:36997
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.2.248.129:41709
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.2.248.130:45487
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.2.248.131:43453
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.2.248.132:36129
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.2.248.133:43177
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.2.248.134:33609
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.2.248.135:36997

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:42939 | all 7 server(s) checked
 time_source         | builtin             | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 08:11:52.893733  3042 log_verifier.cc:126] Checking tablet 217060ad409b4492a7d4ce043578aa49
I20240501 08:11:53.023774  3042 log_verifier.cc:177] Verified matching terms for 328 ops in tablet 217060ad409b4492a7d4ce043578aa49
I20240501 08:11:53.024243  3042 log_verifier.cc:126] Checking tablet 27f9665553394cf4a53e8ebc82fbc575
I20240501 08:11:53.180445  3042 log_verifier.cc:177] Verified matching terms for 340 ops in tablet 27f9665553394cf4a53e8ebc82fbc575
I20240501 08:11:53.181212  3042 log_verifier.cc:126] Checking tablet 366e736f8763465299980fe09c0159bf
I20240501 08:11:53.322356  3042 log_verifier.cc:177] Verified matching terms for 318 ops in tablet 366e736f8763465299980fe09c0159bf
I20240501 08:11:53.322751  3042 log_verifier.cc:126] Checking tablet 4f476973a1f24e09b30075aff39ce31c
I20240501 08:11:53.453145  3042 log_verifier.cc:177] Verified matching terms for 347 ops in tablet 4f476973a1f24e09b30075aff39ce31c
I20240501 08:11:53.453570  3042 log_verifier.cc:126] Checking tablet 527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:11:53.573527  3042 log_verifier.cc:177] Verified matching terms for 302 ops in tablet 527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:11:53.573885  3042 log_verifier.cc:126] Checking tablet 5924bdc199d04ba495059ea3c60f2792
I20240501 08:11:53.723714  3042 log_verifier.cc:177] Verified matching terms for 365 ops in tablet 5924bdc199d04ba495059ea3c60f2792
I20240501 08:11:53.724215  3042 log_verifier.cc:126] Checking tablet 74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:11:53.841943  3042 log_verifier.cc:177] Verified matching terms for 330 ops in tablet 74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:11:53.842401  3042 log_verifier.cc:126] Checking tablet 89727cd975064b5680ff992414dbaef0
I20240501 08:11:53.978287  3042 log_verifier.cc:177] Verified matching terms for 369 ops in tablet 89727cd975064b5680ff992414dbaef0
I20240501 08:11:53.978739  3042 log_verifier.cc:126] Checking tablet 8cc900308d554b08b5e54974c0ae77e9
I20240501 08:11:54.100768  3042 log_verifier.cc:177] Verified matching terms for 310 ops in tablet 8cc900308d554b08b5e54974c0ae77e9
I20240501 08:11:54.101260  3042 log_verifier.cc:126] Checking tablet 8dbcd048e5a94589a1ea17c449688af4
I20240501 08:11:54.228426  3042 log_verifier.cc:177] Verified matching terms for 322 ops in tablet 8dbcd048e5a94589a1ea17c449688af4
I20240501 08:11:54.228838  3042 log_verifier.cc:126] Checking tablet 97da0f9e56f14f6a985b6ffe8c44b68c
I20240501 08:11:54.364279  3042 log_verifier.cc:177] Verified matching terms for 327 ops in tablet 97da0f9e56f14f6a985b6ffe8c44b68c
I20240501 08:11:54.364749  3042 log_verifier.cc:126] Checking tablet 9cc205621c874d5ab791bb3329988ccc
I20240501 08:11:54.494927  3042 log_verifier.cc:177] Verified matching terms for 350 ops in tablet 9cc205621c874d5ab791bb3329988ccc
I20240501 08:11:54.495366  3042 log_verifier.cc:126] Checking tablet a3beb371e6b749ec9580bca95f0908d1
I20240501 08:11:54.624560  3042 log_verifier.cc:177] Verified matching terms for 345 ops in tablet a3beb371e6b749ec9580bca95f0908d1
I20240501 08:11:54.624980  3042 log_verifier.cc:126] Checking tablet b421e60e689c4adfabadb47bd9a9e167
I20240501 08:11:54.644721  3042 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b421e60e689c4adfabadb47bd9a9e167
I20240501 08:11:54.644824  3042 log_verifier.cc:126] Checking tablet dff918bc67204080bf1c272c2d950045
I20240501 08:11:54.775979  3042 log_verifier.cc:177] Verified matching terms for 344 ops in tablet dff918bc67204080bf1c272c2d950045
I20240501 08:11:54.776492  3042 log_verifier.cc:126] Checking tablet f2b0f6861d824903ab421529fdd2e55a
I20240501 08:11:54.909978  3042 log_verifier.cc:177] Verified matching terms for 343 ops in tablet f2b0f6861d824903ab421529fdd2e55a
I20240501 08:11:54.913061  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.136:0
--local_ip_for_outbound_sockets=127.2.248.136
--webserver_interface=127.2.248.136
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--builtin_ntp_servers=127.2.248.148:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:11:55.281740  9454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:55.282500  9454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:55.282634  9454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:55.290335  9454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:55.290469  9454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:55.290624  9454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.136
I20240501 08:11:55.300447  9454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.248.136
--webserver_port=0
--tserver_master_addrs=127.2.248.190:39497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.136
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:55.301807  9454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:55.303534  9454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:55.316378  9459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:55.316951  9462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:55.317469  9460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:55.318178  9454 server_base.cc:1034] running on GCE node
I20240501 08:11:55.318758  9454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:55.319686  9454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:55.320911  9454 hybrid_clock.cc:648] HybridClock initialized: now 1714551115320853 us; error 40 us; skew 500 ppm
I20240501 08:11:55.321250  9454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:55.323617  9454 webserver.cc:460] Webserver started at http://127.2.248.136:40073/ using document root <none> and password file <none>
I20240501 08:11:55.324417  9454 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:55.324532  9454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:55.324839  9454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:55.327606  9454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e5b5d3c836b64c20add9b710808858ba"
format_stamp: "Formatted at 2024-05-01 08:11:55 on dist-test-slave-0pdg"
I20240501 08:11:55.328469  9454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e5b5d3c836b64c20add9b710808858ba"
format_stamp: "Formatted at 2024-05-01 08:11:55 on dist-test-slave-0pdg"
I20240501 08:11:55.334003  9454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240501 08:11:55.337412  9468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:55.338707  9454 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:11:55.338845  9454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e5b5d3c836b64c20add9b710808858ba"
format_stamp: "Formatted at 2024-05-01 08:11:55 on dist-test-slave-0pdg"
I20240501 08:11:55.339046  9454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:55.366286  9454 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:11:55.367254  9454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:55.367583  9454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:55.369235  9454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:55.370734  9454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:55.370831  9454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:55.370945  9454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:55.371017  9454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:11:55.406246  9454 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.136:37495
I20240501 08:11:55.406271  9580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.136:37495 every 8 connection(s)
I20240501 08:11:55.407776  9454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:11:55.412806  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 9454
I20240501 08:11:55.412986  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:11:55.428766  9581 heartbeater.cc:347] Connected to a master server at 127.2.248.190:39497
I20240501 08:11:55.429109  9581 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:55.429903  9581 heartbeater.cc:511] Master 127.2.248.190:39497 requested a full tablet report, sending...
I20240501 08:11:55.431355  6398 ts_manager.cc:194] Registered new tserver with Master: e5b5d3c836b64c20add9b710808858ba (127.2.248.136:37495)
I20240501 08:11:55.432366  6398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.136:48067
I20240501 08:11:55.981581  6522 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:55.983285  6652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:55.984342  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:55.988816  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:55.990435  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:55.997634  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:56.018915  8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:56.039564  9516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:56.245986  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:56.252694  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:56.258566  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:56.262087  8118 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:56.265244  6522 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:56.285645  6652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:56.303925  9516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:56.332718  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:56.406015  6800 consensus_queue.cc:237] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 1.330, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:56.410176  8275 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 330. Preceding OpId from leader: term: 1 index: 331. (index mismatch)
I20240501 08:11:56.410990  9678 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20240501 08:11:56.415094  9362 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.331: config changed from index 70 to 331, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
W20240501 08:11:56.419194  6735 consensus_peers.cc:485] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba -> Peer e5b5d3c836b64c20add9b710808858ba (127.2.248.136:37495): Couldn't send request to peer e5b5d3c836b64c20add9b710808858ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74cea6ae2bd446ce8adc7442dea7f030. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:56.420228  8275 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.331: config changed from index 70 to 331, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
I20240501 08:11:56.424309  6392 catalog_manager.cc:5484] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 70 to 331, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:56.435166  9581 heartbeater.cc:503] Master 127.2.248.190:39497 was elected leader, sending a full tablet report...
I20240501 08:11:56.438427  6670 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:56.441236  8135 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20240501 08:11:56.441881  9693 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240501 08:11:56.446182  9343 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.346: config changed from index 92 to 346, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
I20240501 08:11:56.446642  8135 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 92 to 346, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
I20240501 08:11:56.451313  6398 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: config changed from index 92 to 346, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
W20240501 08:11:56.453855  6605 consensus_peers.cc:485] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 -> Peer e5b5d3c836b64c20add9b710808858ba (127.2.248.136:37495): Couldn't send request to peer e5b5d3c836b64c20add9b710808858ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3beb371e6b749ec9580bca95f0908d1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:56.467777  6541 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:56.470597  8275 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240501 08:11:56.471328  9703 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240501 08:11:56.474356  9001 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 88 to 351, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
I20240501 08:11:56.474874  8275 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 88 to 351, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
I20240501 08:11:56.479326  6398 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 88 to 351, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:56.479935  6475 consensus_peers.cc:485] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d -> Peer e5b5d3c836b64c20add9b710808858ba (127.2.248.136:37495): Couldn't send request to peer e5b5d3c836b64c20add9b710808858ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cc205621c874d5ab791bb3329988ccc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:56.493148  6541 consensus_queue.cc:237] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 1.343, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
W20240501 08:11:56.495824  6475 consensus_peers.cc:485] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d -> Peer e5b5d3c836b64c20add9b710808858ba (127.2.248.136:37495): Couldn't send request to peer e5b5d3c836b64c20add9b710808858ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2b0f6861d824903ab421529fdd2e55a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:56.495868  6800 raft_consensus.cc:1261] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch)
I20240501 08:11:56.496380  9703 consensus_queue.cc:1035] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20240501 08:11:56.499037  9001 raft_consensus.cc:2932] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.344: config changed from index -1 to 344, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
I20240501 08:11:56.499631  6800 raft_consensus.cc:2932] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index -1 to 344, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } } }
I20240501 08:11:56.503760  6392 catalog_manager.cc:5484] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index -1 to 344, NON_VOTER e5b5d3c836b64c20add9b710808858ba (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:56.855573  9754 ts_tablet_manager.cc:902] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: Initiating tablet copy from peer d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487)
I20240501 08:11:56.859462  9754 tablet_copy_client.cc:308] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45487
I20240501 08:11:56.867980  6692 tablet_copy_service.cc:140] P d143a9ab17334a6081de63802bdce654: Received BeginTabletCopySession request for tablet a3beb371e6b749ec9580bca95f0908d1 from peer e5b5d3c836b64c20add9b710808858ba ({username='slave'} at 127.2.248.136:54261)
I20240501 08:11:56.868305  6692 tablet_copy_service.cc:161] P d143a9ab17334a6081de63802bdce654: Beginning new tablet copy session on tablet a3beb371e6b749ec9580bca95f0908d1 from peer e5b5d3c836b64c20add9b710808858ba at {username='slave'} at 127.2.248.136:54261: session id = e5b5d3c836b64c20add9b710808858ba-a3beb371e6b749ec9580bca95f0908d1
I20240501 08:11:56.870676  6692 tablet_copy_source_session.cc:205] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:56.875039  9754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3beb371e6b749ec9580bca95f0908d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.886806  9754 tablet_copy_client.cc:783] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:56.887131  9754 tablet_copy_client.cc:647] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:56.890187  9754 tablet_copy_client.cc:523] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:56.893225  9754 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: Bootstrap starting.
I20240501 08:11:56.904446  9760 ts_tablet_manager.cc:902] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:56.906299  9760 tablet_copy_client.cc:308] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:56.912326  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet 9cc205621c874d5ab791bb3329988ccc from peer e5b5d3c836b64c20add9b710808858ba ({username='slave'} at 127.2.248.136:49701)
I20240501 08:11:56.912504  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet 9cc205621c874d5ab791bb3329988ccc from peer e5b5d3c836b64c20add9b710808858ba at {username='slave'} at 127.2.248.136:49701: session id = e5b5d3c836b64c20add9b710808858ba-9cc205621c874d5ab791bb3329988ccc
I20240501 08:11:56.915463  6562 tablet_copy_source_session.cc:205] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:56.916929  9760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc205621c874d5ab791bb3329988ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.919205  9765 ts_tablet_manager.cc:902] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: Initiating tablet copy from peer 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453)
I20240501 08:11:56.923990  9765 tablet_copy_client.cc:308] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:43453
I20240501 08:11:56.924454  9760 tablet_copy_client.cc:783] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:56.924690  9760 tablet_copy_client.cc:647] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:56.927819  9760 tablet_copy_client.cc:523] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:56.931464  9760 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: Bootstrap starting.
I20240501 08:11:56.932277  6822 tablet_copy_service.cc:140] P 7fb652773c524f639dba8c189b7743ba: Received BeginTabletCopySession request for tablet 74cea6ae2bd446ce8adc7442dea7f030 from peer e5b5d3c836b64c20add9b710808858ba ({username='slave'} at 127.2.248.136:45985)
I20240501 08:11:56.932585  6822 tablet_copy_service.cc:161] P 7fb652773c524f639dba8c189b7743ba: Beginning new tablet copy session on tablet 74cea6ae2bd446ce8adc7442dea7f030 from peer e5b5d3c836b64c20add9b710808858ba at {username='slave'} at 127.2.248.136:45985: session id = e5b5d3c836b64c20add9b710808858ba-74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:11:56.936007  6822 tablet_copy_source_session.cc:205] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:56.937363  9765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cea6ae2bd446ce8adc7442dea7f030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.944454  9765 tablet_copy_client.cc:783] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:56.944813  9765 tablet_copy_client.cc:647] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:56.953114  9770 ts_tablet_manager.cc:902] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: Initiating tablet copy from peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
I20240501 08:11:56.953344  9765 tablet_copy_client.cc:523] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:56.955893  9754 log.cc:824] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:56.958969  9765 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: Bootstrap starting.
I20240501 08:11:56.971179  9770 tablet_copy_client.cc:308] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:41709
I20240501 08:11:56.972137  6562 tablet_copy_service.cc:140] P b0190f14862148f4bde78afbbf1ba43d: Received BeginTabletCopySession request for tablet f2b0f6861d824903ab421529fdd2e55a from peer e5b5d3c836b64c20add9b710808858ba ({username='slave'} at 127.2.248.136:49701)
I20240501 08:11:56.972409  6562 tablet_copy_service.cc:161] P b0190f14862148f4bde78afbbf1ba43d: Beginning new tablet copy session on tablet f2b0f6861d824903ab421529fdd2e55a from peer e5b5d3c836b64c20add9b710808858ba at {username='slave'} at 127.2.248.136:49701: session id = e5b5d3c836b64c20add9b710808858ba-f2b0f6861d824903ab421529fdd2e55a
I20240501 08:11:56.975384  6562 tablet_copy_source_session.cc:205] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:56.976747  9770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b0f6861d824903ab421529fdd2e55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.983518  9770 tablet_copy_client.cc:783] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:56.984000  9770 tablet_copy_client.cc:647] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:56.989603  9770 tablet_copy_client.cc:523] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:56.998608  9770 tablet_bootstrap.cc:492] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: Bootstrap starting.
I20240501 08:11:57.561107  9754 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=346 overwritten=0 applied=346 ignored=0} inserts{seen=1028 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:57.561877  9754 tablet_bootstrap.cc:492] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: Bootstrap complete.
I20240501 08:11:57.562541  9754 ts_tablet_manager.cc:1367] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: Time spent bootstrapping tablet: real 0.670s	user 0.431s	sys 0.033s
I20240501 08:11:57.558247  9760 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=351 overwritten=0 applied=351 ignored=0} inserts{seen=1020 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:57.565941  9760 tablet_bootstrap.cc:492] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: Bootstrap complete.
I20240501 08:11:57.566431  9760 ts_tablet_manager.cc:1367] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: Time spent bootstrapping tablet: real 0.635s	user 0.379s	sys 0.074s
I20240501 08:11:57.572036  9760 raft_consensus.cc:348] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.572580  9760 raft_consensus.cc:729] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5b5d3c836b64c20add9b710808858ba, State: Initialized, Role: LEARNER
I20240501 08:11:57.573211  9760 consensus_queue.cc:260] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 351, Last appended: 1.351, Last appended by leader: 351, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.573923  9754 raft_consensus.cc:348] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.574136  9754 raft_consensus.cc:729] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5b5d3c836b64c20add9b710808858ba, State: Initialized, Role: LEARNER
I20240501 08:11:57.574394  9754 consensus_queue.cc:260] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 346, Last appended: 1.346, Last appended by leader: 346, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.576010  9760 ts_tablet_manager.cc:1398] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20240501 08:11:57.578315  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session e5b5d3c836b64c20add9b710808858ba-9cc205621c874d5ab791bb3329988ccc received from {username='slave'} at 127.2.248.136:49701
I20240501 08:11:57.578454  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session e5b5d3c836b64c20add9b710808858ba-9cc205621c874d5ab791bb3329988ccc on tablet 9cc205621c874d5ab791bb3329988ccc with peer e5b5d3c836b64c20add9b710808858ba
I20240501 08:11:57.590598  9754 ts_tablet_manager.cc:1398] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba: Time spent starting tablet: real 0.028s	user 0.011s	sys 0.003s
I20240501 08:11:57.591462  6692 tablet_copy_service.cc:342] P d143a9ab17334a6081de63802bdce654: Request end of tablet copy session e5b5d3c836b64c20add9b710808858ba-a3beb371e6b749ec9580bca95f0908d1 received from {username='slave'} at 127.2.248.136:54261
I20240501 08:11:57.591625  6692 tablet_copy_service.cc:433] P d143a9ab17334a6081de63802bdce654: ending tablet copy session e5b5d3c836b64c20add9b710808858ba-a3beb371e6b749ec9580bca95f0908d1 on tablet a3beb371e6b749ec9580bca95f0908d1 with peer e5b5d3c836b64c20add9b710808858ba
I20240501 08:11:57.611097  9765 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=331 overwritten=0 applied=331 ignored=0} inserts{seen=959 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:57.611604  9765 tablet_bootstrap.cc:492] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: Bootstrap complete.
I20240501 08:11:57.611917  9765 ts_tablet_manager.cc:1367] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: Time spent bootstrapping tablet: real 0.653s	user 0.410s	sys 0.070s
I20240501 08:11:57.612756  9765 raft_consensus.cc:348] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.612953  9765 raft_consensus.cc:729] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5b5d3c836b64c20add9b710808858ba, State: Initialized, Role: LEARNER
I20240501 08:11:57.613143  9765 consensus_queue.cc:260] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 331, Last appended: 1.331, Last appended by leader: 331, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.613613  9765 ts_tablet_manager.cc:1398] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240501 08:11:57.614835  6822 tablet_copy_service.cc:342] P 7fb652773c524f639dba8c189b7743ba: Request end of tablet copy session e5b5d3c836b64c20add9b710808858ba-74cea6ae2bd446ce8adc7442dea7f030 received from {username='slave'} at 127.2.248.136:45985
I20240501 08:11:57.614972  6822 tablet_copy_service.cc:433] P 7fb652773c524f639dba8c189b7743ba: ending tablet copy session e5b5d3c836b64c20add9b710808858ba-74cea6ae2bd446ce8adc7442dea7f030 on tablet 74cea6ae2bd446ce8adc7442dea7f030 with peer e5b5d3c836b64c20add9b710808858ba
I20240501 08:11:57.650287  9770 tablet_bootstrap.cc:492] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=344 overwritten=0 applied=344 ignored=0} inserts{seen=975 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:57.650668  9770 tablet_bootstrap.cc:492] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: Bootstrap complete.
I20240501 08:11:57.650907  9770 ts_tablet_manager.cc:1367] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: Time spent bootstrapping tablet: real 0.652s	user 0.383s	sys 0.055s
I20240501 08:11:57.651549  9770 raft_consensus.cc:348] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.651759  9770 raft_consensus.cc:729] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5b5d3c836b64c20add9b710808858ba, State: Initialized, Role: LEARNER
I20240501 08:11:57.651902  9770 consensus_queue.cc:260] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 344, Last appended: 1.344, Last appended by leader: 344, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: true } }
I20240501 08:11:57.652370  9770 ts_tablet_manager.cc:1398] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240501 08:11:57.653403  6562 tablet_copy_service.cc:342] P b0190f14862148f4bde78afbbf1ba43d: Request end of tablet copy session e5b5d3c836b64c20add9b710808858ba-f2b0f6861d824903ab421529fdd2e55a received from {username='slave'} at 127.2.248.136:49701
I20240501 08:11:57.653541  6562 tablet_copy_service.cc:433] P b0190f14862148f4bde78afbbf1ba43d: ending tablet copy session e5b5d3c836b64c20add9b710808858ba-f2b0f6861d824903ab421529fdd2e55a on tablet f2b0f6861d824903ab421529fdd2e55a with peer e5b5d3c836b64c20add9b710808858ba
I20240501 08:11:57.817128  9536 raft_consensus.cc:1203] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.345->[1.346-1.346]   Dedup: 1.346->[]
I20240501 08:11:57.858573  9536 raft_consensus.cc:1203] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.343->[1.344-1.344]   Dedup: 1.344->[]
I20240501 08:11:57.878319  9536 raft_consensus.cc:1203] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.330->[1.331-1.331]   Dedup: 1.331->[]
I20240501 08:11:57.999747  9536 raft_consensus.cc:1203] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.350->[1.351-1.351]   Dedup: 1.351->[]
I20240501 08:11:58.235945  9801 raft_consensus.cc:1050] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: attempting to promote NON_VOTER e5b5d3c836b64c20add9b710808858ba to VOTER
I20240501 08:11:58.236644  9801 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 1.346, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.238776  8135 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240501 08:11:58.238940  9536 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Refusing update from remote peer d143a9ab17334a6081de63802bdce654: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240501 08:11:58.239339  9800 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240501 08:11:58.240211  9800 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240501 08:11:58.241895  9802 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Committing config change with OpId 1.347: config changed from index 346 to 347, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.242633  8135 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.247287  9536 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.247633  6392 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: config changed from index 346 to 347, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d143a9ab17334a6081de63802bdce654" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.315156  9803 raft_consensus.cc:1050] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER e5b5d3c836b64c20add9b710808858ba to VOTER
I20240501 08:11:58.315822  9803 consensus_queue.cc:237] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 1.344, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.318606  6800 raft_consensus.cc:1261] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 344. Preceding OpId from leader: term: 1 index: 345. (index mismatch)
I20240501 08:11:58.318693  9536 raft_consensus.cc:1261] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 344. Preceding OpId from leader: term: 1 index: 345. (index mismatch)
I20240501 08:11:58.319275  9803 consensus_queue.cc:1035] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20240501 08:11:58.319939  9804 consensus_queue.cc:1035] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20240501 08:11:58.322641  9803 raft_consensus.cc:2932] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.345: config changed from index 344 to 345, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.323433  6800 raft_consensus.cc:2932] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Committing config change with OpId 1.345: config changed from index 344 to 345, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.327462  9536 raft_consensus.cc:2932] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Committing config change with OpId 1.345: config changed from index 344 to 345, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.328490  6398 catalog_manager.cc:5484] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 344 to 345, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } attrs { replace: true } } peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.336688  6541 consensus_queue.cc:237] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.338721  9536 raft_consensus.cc:1261] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20240501 08:11:58.339200  9803 consensus_queue.cc:1035] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240501 08:11:58.342201  9804 raft_consensus.cc:2932] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.346: config changed from index 345 to 346, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.342962  9536 raft_consensus.cc:2932] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 345 to 346, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.345906  6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2b0f6861d824903ab421529fdd2e55a with cas_config_opid_index 345: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:58.347561  6392 catalog_manager.cc:5484] T f2b0f6861d824903ab421529fdd2e55a P b0190f14862148f4bde78afbbf1ba43d reported cstate change: config changed from index 345 to 346, VOTER 7fb652773c524f639dba8c189b7743ba (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:58.351238  6781 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b0f6861d824903ab421529fdd2e55a with delete_type TABLET_DATA_TOMBSTONED (TS 7fb652773c524f639dba8c189b7743ba not found in new config with opid_index 346) from {username='slave'} at 127.0.0.1:44490
I20240501 08:11:58.363602  9897 tablet_replica.cc:330] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: stopping tablet replica
I20240501 08:11:58.363899  9897 raft_consensus.cc:2229] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:58.364302  9897 raft_consensus.cc:2258] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:58.366180  9897 ts_tablet_manager.cc:1875] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:58.370316  6670 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3beb371e6b749ec9580bca95f0908d1"
dest_uuid: "d143a9ab17334a6081de63802bdce654"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34932
I20240501 08:11:58.370517  6670 raft_consensus.cc:595] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:58.371106  9897 ts_tablet_manager.cc:1888] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.345
I20240501 08:11:58.371254  9897 log.cc:1196] T f2b0f6861d824903ab421529fdd2e55a P 7fb652773c524f639dba8c189b7743ba: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2b0f6861d824903ab421529fdd2e55a
I20240501 08:11:58.372502  6386 catalog_manager.cc:4830] TS 7fb652773c524f639dba8c189b7743ba (127.2.248.131:43453): tablet f2b0f6861d824903ab421529fdd2e55a (table rebalance_test_table_3 [id=a5f969de44c94b85970f9ea504c7804a]) successfully deleted
I20240501 08:11:58.424163  9909 raft_consensus.cc:1050] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba: attempting to promote NON_VOTER e5b5d3c836b64c20add9b710808858ba to VOTER
I20240501 08:11:58.424789  9909 consensus_queue.cc:237] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 1.331, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.426937  9536 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20240501 08:11:58.427060  8275 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20240501 08:11:58.427534  9908 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20240501 08:11:58.427887  9806 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20240501 08:11:58.430625  9908 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.332: config changed from index 331 to 332, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.431242  9536 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 331 to 332, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.432614  8275 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 331 to 332, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.436228  6401 catalog_manager.cc:5484] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 331 to 332, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:58.440202  6800 consensus_queue.cc:237] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 1.332, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.441603  9536 raft_consensus.cc:1261] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Refusing update from remote peer 7fb652773c524f639dba8c189b7743ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20240501 08:11:58.442086  9909 consensus_queue.cc:1035] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20240501 08:11:58.444160  9806 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba [term 1 LEADER]: Committing config change with OpId 1.333: config changed from index 332 to 333, VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.444720  9536 raft_consensus.cc:2932] T 74cea6ae2bd446ce8adc7442dea7f030 P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.447124  6386 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74cea6ae2bd446ce8adc7442dea7f030 with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:58.447993  6392 catalog_manager.cc:5484] T 74cea6ae2bd446ce8adc7442dea7f030 P 7fb652773c524f639dba8c189b7743ba reported cstate change: config changed from index 332 to 333, VOTER ad679e52ba5643f2b11838d14696d1ed (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "7fb652773c524f639dba8c189b7743ba" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "7fb652773c524f639dba8c189b7743ba" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:58.456979  8255 tablet_service.cc:1514] Processing DeleteTablet for tablet 74cea6ae2bd446ce8adc7442dea7f030 with delete_type TABLET_DATA_TOMBSTONED (TS ad679e52ba5643f2b11838d14696d1ed not found in new config with opid_index 333) from {username='slave'} at 127.0.0.1:51192
I20240501 08:11:58.457798  9916 tablet_replica.cc:330] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: stopping tablet replica
I20240501 08:11:58.458171  9916 raft_consensus.cc:2229] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:58.458508  9916 raft_consensus.cc:2258] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:58.460359  9916 ts_tablet_manager.cc:1875] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:58.464588  9916 ts_tablet_manager.cc:1888] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.332
I20240501 08:11:58.464711  9916 log.cc:1196] T 74cea6ae2bd446ce8adc7442dea7f030 P ad679e52ba5643f2b11838d14696d1ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/wals/74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:11:58.465510  6388 catalog_manager.cc:4830] TS ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997): tablet 74cea6ae2bd446ce8adc7442dea7f030 (table rebalance_test_table_4 [id=39aff6d5829146fca6483f16b903dc71]) successfully deleted
I20240501 08:11:58.574434  9804 raft_consensus.cc:1050] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: attempting to promote NON_VOTER e5b5d3c836b64c20add9b710808858ba to VOTER
I20240501 08:11:58.575403  9804 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 1.351, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.577913  8275 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240501 08:11:58.578120  9536 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 1 LEARNER]: Refusing update from remote peer b0190f14862148f4bde78afbbf1ba43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20240501 08:11:58.578589  9804 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240501 08:11:58.578920  9706 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20240501 08:11:58.581995  9804 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Committing config change with OpId 1.352: config changed from index 351 to 352, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.582509  8275 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.587091  6392 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed reported cstate change: config changed from index 351 to 352, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0190f14862148f4bde78afbbf1ba43d" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.596748  9536 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, e5b5d3c836b64c20add9b710808858ba (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:58.611510  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39700
I20240501 08:11:58.611750  6541 raft_consensus.cc:595] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:58.630980  9800 raft_consensus.cc:979] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: : Instructing follower c284970db26b4c95a9ee09310b5adcbe to start an election
I20240501 08:11:58.631142  9800 raft_consensus.cc:1067] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Signalling peer c284970db26b4c95a9ee09310b5adcbe to start an election
I20240501 08:11:58.631767  8135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3beb371e6b749ec9580bca95f0908d1"
dest_uuid: "c284970db26b4c95a9ee09310b5adcbe"
 from {username='slave'} at 127.2.248.130:60569
I20240501 08:11:58.632040  8135 raft_consensus.cc:482] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:58.632266  8135 raft_consensus.cc:3037] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:58.634311  8135 raft_consensus.cc:504] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.635063  8135 leader_election.cc:288] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [CANDIDATE]: Term 2 election: Requested vote from peers d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487), e5b5d3c836b64c20add9b710808858ba (127.2.248.136:37495)
I20240501 08:11:58.635450  6670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3beb371e6b749ec9580bca95f0908d1" candidate_uuid: "c284970db26b4c95a9ee09310b5adcbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 347 } } ignore_live_leader: true dest_uuid: "d143a9ab17334a6081de63802bdce654"
I20240501 08:11:58.635679  6670 raft_consensus.cc:3032] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:58.635739  6670 raft_consensus.cc:729] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d143a9ab17334a6081de63802bdce654, State: Running, Role: LEADER
I20240501 08:11:58.635967  6670 consensus_queue.cc:260] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 1.347, Last appended by leader: 347, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.636269  6670 raft_consensus.cc:3037] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:58.638085  6670 raft_consensus.cc:2454] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c284970db26b4c95a9ee09310b5adcbe in term 2.
I20240501 08:11:58.638655  8072 leader_election.cc:302] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c284970db26b4c95a9ee09310b5adcbe, d143a9ab17334a6081de63802bdce654; no voters: 
I20240501 08:11:58.638998  9776 raft_consensus.cc:2781] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:58.640143  9776 raft_consensus.cc:686] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 2 LEADER]: Becoming Leader. State: Replica: c284970db26b4c95a9ee09310b5adcbe, State: Running, Role: LEADER
I20240501 08:11:58.640487  9776 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 1.347, Last appended by leader: 347, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:58.643517  6392 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe reported cstate change: term changed from 1 to 2, leader changed from d143a9ab17334a6081de63802bdce654 (127.2.248.130) to c284970db26b4c95a9ee09310b5adcbe (127.2.248.134). New cstate: current_term: 2 leader_uuid: "c284970db26b4c95a9ee09310b5adcbe" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:58.653304  6670 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3beb371e6b749ec9580bca95f0908d1"
dest_uuid: "d143a9ab17334a6081de63802bdce654"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34940
I20240501 08:11:58.653523  6670 raft_consensus.cc:595] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Received request to transfer leadership
I20240501 08:11:58.653618  6670 raft_consensus.cc:603] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 08:11:58.654103  9536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3beb371e6b749ec9580bca95f0908d1" candidate_uuid: "c284970db26b4c95a9ee09310b5adcbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 347 } } ignore_live_leader: true dest_uuid: "e5b5d3c836b64c20add9b710808858ba"
I20240501 08:11:58.654414  9536 raft_consensus.cc:3037] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:58.657606  9536 raft_consensus.cc:2454] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c284970db26b4c95a9ee09310b5adcbe in term 2.
I20240501 08:11:58.673816  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39722
I20240501 08:11:58.674058  6541 raft_consensus.cc:595] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:58.905408  6541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc"
dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:39746
I20240501 08:11:58.905632  6541 raft_consensus.cc:595] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:59.034066  9803 raft_consensus.cc:979] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: : Instructing follower e5b5d3c836b64c20add9b710808858ba to start an election
I20240501 08:11:59.034269  9803 raft_consensus.cc:1067] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Signalling peer e5b5d3c836b64c20add9b710808858ba to start an election
I20240501 08:11:59.034950  9536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc"
dest_uuid: "e5b5d3c836b64c20add9b710808858ba"
 from {username='slave'} at 127.2.248.129:54399
I20240501 08:11:59.035264  9536 raft_consensus.cc:482] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:59.035383  9536 raft_consensus.cc:3037] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:59.037737  9536 raft_consensus.cc:504] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:59.039047  9536 leader_election.cc:288] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [CANDIDATE]: Term 2 election: Requested vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709), ad679e52ba5643f2b11838d14696d1ed (127.2.248.135:36997)
I20240501 08:11:59.039494  6541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc" candidate_uuid: "e5b5d3c836b64c20add9b710808858ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 352 } } ignore_live_leader: true dest_uuid: "b0190f14862148f4bde78afbbf1ba43d"
I20240501 08:11:59.039762  6541 raft_consensus.cc:3032] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:59.039855  6541 raft_consensus.cc:729] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b0190f14862148f4bde78afbbf1ba43d, State: Running, Role: LEADER
I20240501 08:11:59.040222  6541 consensus_queue.cc:260] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 352, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:59.040565  6541 raft_consensus.cc:3037] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:59.042527  6541 raft_consensus.cc:2454] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5b5d3c836b64c20add9b710808858ba in term 2.
I20240501 08:11:59.043038  9469 leader_election.cc:302] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0190f14862148f4bde78afbbf1ba43d, e5b5d3c836b64c20add9b710808858ba; no voters: 
I20240501 08:11:59.043426  9825 raft_consensus.cc:2781] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:59.043978  9825 raft_consensus.cc:686] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 2 LEADER]: Becoming Leader. State: Replica: e5b5d3c836b64c20add9b710808858ba, State: Running, Role: LEADER
I20240501 08:11:59.044469  9825 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 352, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:59.046465  8275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc205621c874d5ab791bb3329988ccc" candidate_uuid: "e5b5d3c836b64c20add9b710808858ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 352 } } ignore_live_leader: true dest_uuid: "ad679e52ba5643f2b11838d14696d1ed"
I20240501 08:11:59.046686  8275 raft_consensus.cc:3037] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:59.048807  6392 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba reported cstate change: term changed from 1 to 2, leader changed from b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) to e5b5d3c836b64c20add9b710808858ba (127.2.248.136). New cstate: current_term: 2 leader_uuid: "e5b5d3c836b64c20add9b710808858ba" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:59.049335  8275 raft_consensus.cc:2454] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5b5d3c836b64c20add9b710808858ba in term 2.
I20240501 08:11:59.090198  9536 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 2 FOLLOWER]: Refusing update from remote peer c284970db26b4c95a9ee09310b5adcbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20240501 08:11:59.090870  9776 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20240501 08:11:59.101315  6670 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Refusing update from remote peer c284970db26b4c95a9ee09310b5adcbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20240501 08:11:59.101840  9946 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d143a9ab17334a6081de63802bdce654" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20240501 08:11:59.103909  8135 consensus_queue.cc:237] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 2.348, Last appended by leader: 347, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:11:59.105254  9536 raft_consensus.cc:1261] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 2 FOLLOWER]: Refusing update from remote peer c284970db26b4c95a9ee09310b5adcbe: Log matching property violated. Preceding OpId in replica: term: 2 index: 348. Preceding OpId from leader: term: 2 index: 349. (index mismatch)
I20240501 08:11:59.105752  9946 consensus_queue.cc:1035] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240501 08:11:59.108100  9776 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P c284970db26b4c95a9ee09310b5adcbe [term 2 LEADER]: Committing config change with OpId 2.349: config changed from index 347 to 349, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:59.108759  9536 raft_consensus.cc:2932] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba [term 2 FOLLOWER]: Committing config change with OpId 2.349: config changed from index 347 to 349, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:59.112751  6392 catalog_manager.cc:5484] T a3beb371e6b749ec9580bca95f0908d1 P e5b5d3c836b64c20add9b710808858ba reported cstate change: config changed from index 347 to 349, VOTER d143a9ab17334a6081de63802bdce654 (127.2.248.130) evicted. New cstate: current_term: 2 leader_uuid: "c284970db26b4c95a9ee09310b5adcbe" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "c284970db26b4c95a9ee09310b5adcbe" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 33609 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:11:59.114444  6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3beb371e6b749ec9580bca95f0908d1 with cas_config_opid_index 347: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:59.116720  6652 tablet_service.cc:1514] Processing DeleteTablet for tablet a3beb371e6b749ec9580bca95f0908d1 with delete_type TABLET_DATA_TOMBSTONED (TS d143a9ab17334a6081de63802bdce654 not found in new config with opid_index 349) from {username='slave'} at 127.0.0.1:54172
I20240501 08:11:59.120226  9992 tablet_replica.cc:330] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: stopping tablet replica
I20240501 08:11:59.120537  9992 raft_consensus.cc:2229] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:59.120757  9992 raft_consensus.cc:2258] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:59.122613  9992 ts_tablet_manager.cc:1875] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:59.127434  9992 ts_tablet_manager.cc:1888] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.348
I20240501 08:11:59.127547  9992 log.cc:1196] T a3beb371e6b749ec9580bca95f0908d1 P d143a9ab17334a6081de63802bdce654: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3beb371e6b749ec9580bca95f0908d1
I20240501 08:11:59.128350  6386 catalog_manager.cc:4830] TS d143a9ab17334a6081de63802bdce654 (127.2.248.130:45487): tablet a3beb371e6b749ec9580bca95f0908d1 (table rebalance_test_table_0 [id=585c1bb498bf42bea9e424b9373c5233]) successfully deleted
I20240501 08:11:59.476716  6541 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Refusing update from remote peer e5b5d3c836b64c20add9b710808858ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20240501 08:11:59.477501  9825 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240501 08:11:59.482026  8275 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Refusing update from remote peer e5b5d3c836b64c20add9b710808858ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20240501 08:11:59.482661  9825 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240501 08:12:00.053191  9536 consensus_queue.cc:237] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 352, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:12:00.054693  8275 raft_consensus.cc:1261] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Refusing update from remote peer e5b5d3c836b64c20add9b710808858ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 2 index: 354. (index mismatch)
I20240501 08:12:00.055166 10034 consensus_queue.cc:1035] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240501 08:12:00.060560 10033 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba [term 2 LEADER]: Committing config change with OpId 2.354: config changed from index 352 to 354, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:12:00.061177  8275 raft_consensus.cc:2932] T 9cc205621c874d5ab791bb3329988ccc P ad679e52ba5643f2b11838d14696d1ed [term 2 FOLLOWER]: Committing config change with OpId 2.354: config changed from index 352 to 354, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } } }
I20240501 08:12:00.065706  6385 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9cc205621c874d5ab791bb3329988ccc with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:00.067101  6398 catalog_manager.cc:5484] T 9cc205621c874d5ab791bb3329988ccc P e5b5d3c836b64c20add9b710808858ba reported cstate change: config changed from index 352 to 354, VOTER b0190f14862148f4bde78afbbf1ba43d (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "e5b5d3c836b64c20add9b710808858ba" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ad679e52ba5643f2b11838d14696d1ed" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 36997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:00.071769  6522 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cc205621c874d5ab791bb3329988ccc with delete_type TABLET_DATA_TOMBSTONED (TS b0190f14862148f4bde78afbbf1ba43d not found in new config with opid_index 354) from {username='slave'} at 127.0.0.1:51260
I20240501 08:12:00.072502 10040 tablet_replica.cc:330] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: stopping tablet replica
I20240501 08:12:00.072711 10040 raft_consensus.cc:2229] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:00.072925 10040 raft_consensus.cc:2258] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:00.074349 10040 ts_tablet_manager.cc:1875] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:00.078608 10040 ts_tablet_manager.cc:1888] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.353
I20240501 08:12:00.078737 10040 log.cc:1196] T 9cc205621c874d5ab791bb3329988ccc P b0190f14862148f4bde78afbbf1ba43d: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/9cc205621c874d5ab791bb3329988ccc
I20240501 08:12:00.079526  6385 catalog_manager.cc:4830] TS b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709): tablet 9cc205621c874d5ab791bb3329988ccc (table rebalance_test_table_2 [id=ebbfb2db518a491da7441bb6006fe712]) successfully deleted
I20240501 08:12:00.299777  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:00.309593  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:00.325691  6522 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:00.354727  6652 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:00.361248  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:00.368573  8118 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:00.381719  9516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:00.405248  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:00.562935  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:00.567638  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:00.569052  6522 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:00.569804  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:00.589323  6652 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:00.597677  9516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:00.597699  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:00.605613  8118 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:00.925223  6781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:00.943588  7986 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:00.944401  8255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:00.954121  6652 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:00.961102  6522 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:00.964670  8118 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:00.975409  9516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:01.033589  7854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 42d8139962d445eba456f56990374af2 | 127.2.248.190:39497 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:42939 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 0257c0e8038d4e3fa9580cb47278f16f | 127.2.248.133:43177 | HEALTHY | <none>   |       2        |       0
 7fb652773c524f639dba8c189b7743ba | 127.2.248.131:43453 | HEALTHY | <none>   |       3        |       0
 ad679e52ba5643f2b11838d14696d1ed | 127.2.248.135:36997 | HEALTHY | <none>   |       0        |       0
 b0190f14862148f4bde78afbbf1ba43d | 127.2.248.129:41709 | HEALTHY | <none>   |       4        |       0
 c284970db26b4c95a9ee09310b5adcbe | 127.2.248.134:33609 | HEALTHY | <none>   |       2        |       0
 d143a9ab17334a6081de63802bdce654 | 127.2.248.130:45487 | HEALTHY | <none>   |       3        |       0
 e5b5d3c836b64c20add9b710808858ba | 127.2.248.136:37495 | HEALTHY | <none>   |       1        |       0
 fcfccbafc3254a87896f0e72c8078ab5 | 127.2.248.132:36129 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.2.248.129                                                                                                                                                   | experimental          | 127.2.248.129:41709
 local_ip_for_outbound_sockets            | 127.2.248.130                                                                                                                                                   | experimental          | 127.2.248.130:45487
 local_ip_for_outbound_sockets            | 127.2.248.131                                                                                                                                                   | experimental          | 127.2.248.131:43453
 local_ip_for_outbound_sockets            | 127.2.248.132                                                                                                                                                   | experimental          | 127.2.248.132:36129
 local_ip_for_outbound_sockets            | 127.2.248.133                                                                                                                                                   | experimental          | 127.2.248.133:43177
 local_ip_for_outbound_sockets            | 127.2.248.134                                                                                                                                                   | experimental          | 127.2.248.134:33609
 local_ip_for_outbound_sockets            | 127.2.248.135                                                                                                                                                   | experimental          | 127.2.248.135:36997
 local_ip_for_outbound_sockets            | 127.2.248.136                                                                                                                                                   | experimental          | 127.2.248.136:37495
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.2.248.129:41709
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.2.248.130:45487
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.2.248.131:43453
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.2.248.132:36129
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.2.248.133:43177
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.2.248.134:33609
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.2.248.135:36997
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.2.248.136:37495

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:42939 | all 8 server(s) checked
 time_source         | builtin             | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 08:12:01.103775  3042 log_verifier.cc:126] Checking tablet 217060ad409b4492a7d4ce043578aa49
I20240501 08:12:01.249073  3042 log_verifier.cc:177] Verified matching terms for 328 ops in tablet 217060ad409b4492a7d4ce043578aa49
I20240501 08:12:01.249472  3042 log_verifier.cc:126] Checking tablet 27f9665553394cf4a53e8ebc82fbc575
I20240501 08:12:01.389873  3042 log_verifier.cc:177] Verified matching terms for 340 ops in tablet 27f9665553394cf4a53e8ebc82fbc575
I20240501 08:12:01.390338  3042 log_verifier.cc:126] Checking tablet 366e736f8763465299980fe09c0159bf
I20240501 08:12:01.535588  3042 log_verifier.cc:177] Verified matching terms for 318 ops in tablet 366e736f8763465299980fe09c0159bf
I20240501 08:12:01.535977  3042 log_verifier.cc:126] Checking tablet 4f476973a1f24e09b30075aff39ce31c
I20240501 08:12:01.653153  3042 log_verifier.cc:177] Verified matching terms for 347 ops in tablet 4f476973a1f24e09b30075aff39ce31c
I20240501 08:12:01.653575  3042 log_verifier.cc:126] Checking tablet 527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:12:01.769454  3042 log_verifier.cc:177] Verified matching terms for 302 ops in tablet 527072daf1ea4b478a1b35f4a5c4bf3c
I20240501 08:12:01.769855  3042 log_verifier.cc:126] Checking tablet 5924bdc199d04ba495059ea3c60f2792
I20240501 08:12:01.893390  3042 log_verifier.cc:177] Verified matching terms for 365 ops in tablet 5924bdc199d04ba495059ea3c60f2792
I20240501 08:12:01.893849  3042 log_verifier.cc:126] Checking tablet 74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:12:02.010390  3042 log_verifier.cc:177] Verified matching terms for 333 ops in tablet 74cea6ae2bd446ce8adc7442dea7f030
I20240501 08:12:02.010820  3042 log_verifier.cc:126] Checking tablet 89727cd975064b5680ff992414dbaef0
I20240501 08:12:02.152770  3042 log_verifier.cc:177] Verified matching terms for 369 ops in tablet 89727cd975064b5680ff992414dbaef0
I20240501 08:12:02.153229  3042 log_verifier.cc:126] Checking tablet 8cc900308d554b08b5e54974c0ae77e9
I20240501 08:12:02.274957  3042 log_verifier.cc:177] Verified matching terms for 310 ops in tablet 8cc900308d554b08b5e54974c0ae77e9
I20240501 08:12:02.275439  3042 log_verifier.cc:126] Checking tablet 8dbcd048e5a94589a1ea17c449688af4
I20240501 08:12:02.390069  3042 log_verifier.cc:177] Verified matching terms for 322 ops in tablet 8dbcd048e5a94589a1ea17c449688af4
I20240501 08:12:02.390522  3042 log_verifier.cc:126] Checking tablet 97da0f9e56f14f6a985b6ffe8c44b68c
I20240501 08:12:02.511004  3042 log_verifier.cc:177] Verified matching terms for 327 ops in tablet 97da0f9e56f14f6a985b6ffe8c44b68c
I20240501 08:12:02.511446  3042 log_verifier.cc:126] Checking tablet 9cc205621c874d5ab791bb3329988ccc
I20240501 08:12:02.638366  3042 log_verifier.cc:177] Verified matching terms for 354 ops in tablet 9cc205621c874d5ab791bb3329988ccc
I20240501 08:12:02.650007  3042 log_verifier.cc:126] Checking tablet a3beb371e6b749ec9580bca95f0908d1
I20240501 08:12:02.787050  3042 log_verifier.cc:177] Verified matching terms for 349 ops in tablet a3beb371e6b749ec9580bca95f0908d1
I20240501 08:12:02.787494  3042 log_verifier.cc:126] Checking tablet b421e60e689c4adfabadb47bd9a9e167
I20240501 08:12:02.798306  3042 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b421e60e689c4adfabadb47bd9a9e167
I20240501 08:12:02.798385  3042 log_verifier.cc:126] Checking tablet dff918bc67204080bf1c272c2d950045
I20240501 08:12:02.915720  3042 log_verifier.cc:177] Verified matching terms for 344 ops in tablet dff918bc67204080bf1c272c2d950045
I20240501 08:12:02.916218  3042 log_verifier.cc:126] Checking tablet f2b0f6861d824903ab421529fdd2e55a
I20240501 08:12:03.037300  3042 log_verifier.cc:177] Verified matching terms for 346 ops in tablet f2b0f6861d824903ab421529fdd2e55a
I20240501 08:12:03.046154  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 6460
I20240501 08:12:03.047297  6581 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:03.287209  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 6460
W20240501 08:12:03.325008  8209 connection.cc:535] server connection from 127.2.248.129:48679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:12:03.325701  6606 connection.cc:535] server connection from 127.2.248.129:41007 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:03.326416  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 6590
I20240501 08:12:03.327293  6712 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:03.538798  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 6590
I20240501 08:12:03.576109  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 6720
I20240501 08:12:03.577147  6842 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:03.781309  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 6720
W20240501 08:12:03.817751  7809 connection.cc:535] server connection from 127.2.248.131:52013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:03.818295  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 7791
I20240501 08:12:03.830152  7913 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:03.982156  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 7791
I20240501 08:12:04.009188  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 7923
I20240501 08:12:04.015944  8047 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:12:04.084308  8073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.132:36129: connect: Connection refused (error 111)
W20240501 08:12:04.086030  8073 consensus_peers.cc:485] T 217060ad409b4492a7d4ce043578aa49 P c284970db26b4c95a9ee09310b5adcbe -> Peer fcfccbafc3254a87896f0e72c8078ab5 (127.2.248.132:36129): Couldn't send request to peer fcfccbafc3254a87896f0e72c8078ab5. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.132:36129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:04.164681  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 7923
W20240501 08:12:04.192759  8211 connection.cc:535] server connection from 127.2.248.133:50911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:04.193111  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 8056
I20240501 08:12:04.199103  8178 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:04.352411  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 8056
W20240501 08:12:04.379153  9470 connection.cc:535] server connection from 127.2.248.134:55109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:04.379442  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 8186
I20240501 08:12:04.384845  8315 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:04.464875 10168 raft_consensus.cc:482] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0190f14862148f4bde78afbbf1ba43d)
I20240501 08:12:04.465096 10168 raft_consensus.cc:504] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "b0190f14862148f4bde78afbbf1ba43d" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 41709 } } peers { permanent_uuid: "e5b5d3c836b64c20add9b710808858ba" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 37495 } attrs { promote: false } }
I20240501 08:12:04.465730 10168 leader_election.cc:288] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709)
W20240501 08:12:04.466821  9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.129:41709: connect: Connection refused (error 111)
W20240501 08:12:04.468386  9469 leader_election.cc:334] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0190f14862148f4bde78afbbf1ba43d (127.2.248.129:41709): Network error: Client connection negotiation failed: client connection to 127.2.248.129:41709: connect: Connection refused (error 111)
I20240501 08:12:04.468528  9469 leader_election.cc:302] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e5b5d3c836b64c20add9b710808858ba; no voters: b0190f14862148f4bde78afbbf1ba43d
I20240501 08:12:04.468820 10168 raft_consensus.cc:2726] T f2b0f6861d824903ab421529fdd2e55a P e5b5d3c836b64c20add9b710808858ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:12:04.532334  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 8186
I20240501 08:12:04.558423  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 9454
I20240501 08:12:04.563803  9576 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:04.681762  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 9454
I20240501 08:12:04.702306  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 6370
I20240501 08:12:04.703284  6430 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:04.856426  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 6370
2024-05-01T08:12:04Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/3 (48702 ms)
[----------] 1 test from RebalanceParamTest (48702 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240501 08:12:04.942273  3042 test_util.cc:268] Using random seed: 180379058
I20240501 08:12:04.943892  3042 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:12:04.943938  3042 ts_itest-base.cc:113] --------------
I20240501 08:12:04.943961  3042 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:12:04.943995  3042 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:12:04.944028  3042 ts_itest-base.cc:116] --------------
2024-05-01T08:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:12:04Z Disabled control of system clock
I20240501 08:12:04.985514  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:33713
--webserver_interface=127.2.248.190
--webserver_port=0
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.248.190:33713
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:12:05.294921 10190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:05.295194 10190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:05.295269 10190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:05.301789 10190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:05.301882 10190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:12:05.301931 10190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:05.301981 10190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:12:05.302021 10190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:12:05.310210 10190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.248.190:33713
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.248.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:05.311336 10190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:05.312927 10190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:05.323339 10195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:05.323859 10196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:05.324405 10198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:05.325130 10190 server_base.cc:1034] running on GCE node
I20240501 08:12:05.325868 10190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:05.327319 10190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:05.328531 10190 hybrid_clock.cc:648] HybridClock initialized: now 1714551125328490 us; error 27 us; skew 500 ppm
I20240501 08:12:05.328883 10190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:05.331079 10190 webserver.cc:460] Webserver started at http://127.2.248.190:40217/ using document root <none> and password file <none>
I20240501 08:12:05.331779 10190 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:05.331890 10190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:05.332235 10190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:05.334635 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ee59fa3365af4b91907d4ab87092780e"
format_stamp: "Formatted at 2024-05-01 08:12:05 on dist-test-slave-0pdg"
I20240501 08:12:05.335364 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ee59fa3365af4b91907d4ab87092780e"
format_stamp: "Formatted at 2024-05-01 08:12:05 on dist-test-slave-0pdg"
I20240501 08:12:05.340075 10190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.006s
I20240501 08:12:05.343294 10204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:05.344535 10190 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:05.344677 10190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ee59fa3365af4b91907d4ab87092780e"
format_stamp: "Formatted at 2024-05-01 08:12:05 on dist-test-slave-0pdg"
I20240501 08:12:05.344863 10190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:05.363188 10190 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:05.363890 10190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:05.364149 10190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:05.380321 10190 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.190:33713
I20240501 08:12:05.380331 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.190:33713 every 8 connection(s)
I20240501 08:12:05.381512 10190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:12:05.384763 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.387071  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 10190
I20240501 08:12:05.387274  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:12:05.396087 10256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e: Bootstrap starting.
I20240501 08:12:05.399485 10256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.400697 10256 log.cc:824] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:05.402935 10256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e: No bootstrap required, opened a new log
I20240501 08:12:05.408768 10256 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee59fa3365af4b91907d4ab87092780e" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 33713 } }
I20240501 08:12:05.409051 10256 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.409121 10256 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee59fa3365af4b91907d4ab87092780e, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.409562 10256 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee59fa3365af4b91907d4ab87092780e" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 33713 } }
I20240501 08:12:05.409715 10256 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:12:05.409801 10256 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:12:05.409955 10256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.412041 10256 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee59fa3365af4b91907d4ab87092780e" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 33713 } }
I20240501 08:12:05.412516 10256 leader_election.cc:302] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee59fa3365af4b91907d4ab87092780e; no voters: 
I20240501 08:12:05.413013 10256 leader_election.cc:288] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:12:05.413143 10261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:05.413749 10261 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [term 1 LEADER]: Becoming Leader. State: Replica: ee59fa3365af4b91907d4ab87092780e, State: Running, Role: LEADER
I20240501 08:12:05.414249 10261 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee59fa3365af4b91907d4ab87092780e" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 33713 } }
I20240501 08:12:05.414702 10256 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:12:05.417464 10262 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee59fa3365af4b91907d4ab87092780e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee59fa3365af4b91907d4ab87092780e" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 33713 } } }
I20240501 08:12:05.417485 10263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee59fa3365af4b91907d4ab87092780e. Latest consensus state: current_term: 1 leader_uuid: "ee59fa3365af4b91907d4ab87092780e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee59fa3365af4b91907d4ab87092780e" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 33713 } } }
I20240501 08:12:05.417891 10263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:05.417896 10262 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:05.421028 10268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:12:05.426422 10268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:12:05.433760 10268 catalog_manager.cc:1347] Generated new cluster ID: d92fd24dc5fc446b9f9d5dac5049b971
I20240501 08:12:05.433851 10268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:12:05.445999 10268 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:12:05.446972 10268 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:12:05.453467 10268 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee59fa3365af4b91907d4ab87092780e: Generated new TSK 0
I20240501 08:12:05.454077 10268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:12:05.468619  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.129:0
--local_ip_for_outbound_sockets=127.2.248.129
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:12:05.780978 10280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:05.781237 10280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:05.781349 10280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:05.788043 10280 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:05.788168 10280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:05.788336 10280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.129
I20240501 08:12:05.796713 10280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.129
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:05.797873 10280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:05.799367 10280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:05.810245 10286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:05.810720 10288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:05.811177 10285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:05.811853 10280 server_base.cc:1034] running on GCE node
I20240501 08:12:05.812417 10280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:05.813192 10280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:05.814371 10280 hybrid_clock.cc:648] HybridClock initialized: now 1714551125814326 us; error 41 us; skew 500 ppm
I20240501 08:12:05.814639 10280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:05.816746 10280 webserver.cc:460] Webserver started at http://127.2.248.129:34141/ using document root <none> and password file <none>
I20240501 08:12:05.817375 10280 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:05.817481 10280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:05.817759 10280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:05.820019 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c3747ec2a93c4684962f215616e896ed"
format_stamp: "Formatted at 2024-05-01 08:12:05 on dist-test-slave-0pdg"
I20240501 08:12:05.820714 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c3747ec2a93c4684962f215616e896ed"
format_stamp: "Formatted at 2024-05-01 08:12:05 on dist-test-slave-0pdg"
I20240501 08:12:05.824998 10280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 08:12:05.827910 10294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:05.829190 10280 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240501 08:12:05.829334 10280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c3747ec2a93c4684962f215616e896ed"
format_stamp: "Formatted at 2024-05-01 08:12:05 on dist-test-slave-0pdg"
I20240501 08:12:05.829501 10280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:05.852206 10280 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:05.852855 10280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:05.853078 10280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:05.853891 10280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:05.855195 10280 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:05.855336 10280 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:05.855427 10280 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:05.855481 10280 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:05.885495 10280 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.129:44049
I20240501 08:12:05.885536 10406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.129:44049 every 8 connection(s)
I20240501 08:12:05.886695 10280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:12:05.887101  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 10280
I20240501 08:12:05.887279  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:12:05.890717  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.130:0
--local_ip_for_outbound_sockets=127.2.248.130
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:05.899169 10407 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:05.899462 10407 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:05.900434 10407 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:05.902415 10221 ts_manager.cc:194] Registered new tserver with Master: c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:05.903805 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.129:48077
W20240501 08:12:06.217571 10410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:06.217828 10410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:06.217965 10410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:06.224862 10410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:06.224970 10410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:06.225132 10410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.130
I20240501 08:12:06.233592 10410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.130
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:06.234810 10410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:06.236454 10410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:06.247803 10415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:06.248349 10418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:06.248898 10416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:06.249578 10410 server_base.cc:1034] running on GCE node
I20240501 08:12:06.250116 10410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:06.250945 10410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:06.252205 10410 hybrid_clock.cc:648] HybridClock initialized: now 1714551126252155 us; error 43 us; skew 500 ppm
I20240501 08:12:06.252487 10410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:06.254539 10410 webserver.cc:460] Webserver started at http://127.2.248.130:38071/ using document root <none> and password file <none>
I20240501 08:12:06.255256 10410 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:06.255385 10410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:06.255717 10410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:06.258364 10410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "861f629e2d134ff08f25fa553dfb5bda"
format_stamp: "Formatted at 2024-05-01 08:12:06 on dist-test-slave-0pdg"
I20240501 08:12:06.259166 10410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "861f629e2d134ff08f25fa553dfb5bda"
format_stamp: "Formatted at 2024-05-01 08:12:06 on dist-test-slave-0pdg"
I20240501 08:12:06.264252 10410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20240501 08:12:06.267477 10424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:06.268821 10410 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:06.268966 10410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "861f629e2d134ff08f25fa553dfb5bda"
format_stamp: "Formatted at 2024-05-01 08:12:06 on dist-test-slave-0pdg"
I20240501 08:12:06.269269 10410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:06.291523 10410 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:06.292423 10410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:06.292716 10410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:06.293607 10410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:06.294991 10410 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:06.295099 10410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:06.295218 10410 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:06.295289 10410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:06.325976 10410 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.130:46605
I20240501 08:12:06.326021 10536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.130:46605 every 8 connection(s)
I20240501 08:12:06.327189 10410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:12:06.330669  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 10410
I20240501 08:12:06.330843  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:12:06.334296  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:0
--local_ip_for_outbound_sockets=127.2.248.131
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:06.339800 10537 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:06.340130 10537 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:06.341048 10537 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:06.342602 10221 ts_manager.cc:194] Registered new tserver with Master: 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:06.343470 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.130:52217
W20240501 08:12:06.648715 10540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:06.649003 10540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:06.649153 10540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:06.656389 10540 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:06.656533 10540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:06.656693 10540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:12:06.665294 10540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:06.666522 10540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:06.668259 10540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:06.679298 10545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:06.679775 10548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:06.680410 10546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:06.680955 10540 server_base.cc:1034] running on GCE node
I20240501 08:12:06.681463 10540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:06.682265 10540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:06.683444 10540 hybrid_clock.cc:648] HybridClock initialized: now 1714551126683412 us; error 27 us; skew 500 ppm
I20240501 08:12:06.683709 10540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:06.685768 10540 webserver.cc:460] Webserver started at http://127.2.248.131:42107/ using document root <none> and password file <none>
I20240501 08:12:06.686447 10540 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:06.686545 10540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:06.686843 10540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:06.689240 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
format_stamp: "Formatted at 2024-05-01 08:12:06 on dist-test-slave-0pdg"
I20240501 08:12:06.689991 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
format_stamp: "Formatted at 2024-05-01 08:12:06 on dist-test-slave-0pdg"
I20240501 08:12:06.694697 10540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 08:12:06.697759 10554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:06.698935 10540 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:06.699100 10540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
format_stamp: "Formatted at 2024-05-01 08:12:06 on dist-test-slave-0pdg"
I20240501 08:12:06.699278 10540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:06.735041 10540 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:06.735731 10540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:06.735927 10540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:06.736836 10540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:06.738185 10540 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:06.738265 10540 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:06.738348 10540 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:06.738400 10540 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:06.768142 10540 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:34319
I20240501 08:12:06.768205 10666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:34319 every 8 connection(s)
I20240501 08:12:06.769250 10540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:12:06.770694  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 10540
I20240501 08:12:06.770852  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:12:06.774171  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:0
--local_ip_for_outbound_sockets=127.2.248.132
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:06.782841 10667 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:06.783131 10667 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:06.783987 10667 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:06.785174 10221 ts_manager.cc:194] Registered new tserver with Master: 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:06.786171 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:58183
I20240501 08:12:06.906134 10407 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
W20240501 08:12:07.087466 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:07.087733 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:07.087867 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:07.094358 10670 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:07.094485 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:07.094630 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:12:07.102559 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:07.103624 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:07.105194 10670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:07.116055 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:07.116587 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:07.117072 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:07.117781 10670 server_base.cc:1034] running on GCE node
I20240501 08:12:07.118283 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:07.119096 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:07.120333 10670 hybrid_clock.cc:648] HybridClock initialized: now 1714551127120275 us; error 29 us; skew 500 ppm
I20240501 08:12:07.120608 10670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:07.122690 10670 webserver.cc:460] Webserver started at http://127.2.248.132:45189/ using document root <none> and password file <none>
I20240501 08:12:07.123325 10670 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:07.123435 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:07.123741 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:07.126169 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "06c1439c9ab8415485eff20566811ede"
format_stamp: "Formatted at 2024-05-01 08:12:07 on dist-test-slave-0pdg"
I20240501 08:12:07.126964 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "06c1439c9ab8415485eff20566811ede"
format_stamp: "Formatted at 2024-05-01 08:12:07 on dist-test-slave-0pdg"
I20240501 08:12:07.131713 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 08:12:07.134943 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:07.136358 10670 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:07.136512 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "06c1439c9ab8415485eff20566811ede"
format_stamp: "Formatted at 2024-05-01 08:12:07 on dist-test-slave-0pdg"
I20240501 08:12:07.136701 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:07.174224 10670 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:07.175144 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:07.175411 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:07.176414 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:07.177881 10670 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:07.177973 10670 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:07.178059 10670 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:07.178218 10670 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:07.208493 10670 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:40093
I20240501 08:12:07.208582 10796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:40093 every 8 connection(s)
I20240501 08:12:07.209764 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:12:07.211527  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 10670
I20240501 08:12:07.211683  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:12:07.214605  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:0
--local_ip_for_outbound_sockets=127.2.248.133
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:07.221993 10797 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:07.222267 10797 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:07.223026 10797 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:07.224269 10221 ts_manager.cc:194] Registered new tserver with Master: 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:07.225283 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:58241
I20240501 08:12:07.346326 10537 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
W20240501 08:12:07.535625 10800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:07.535883 10800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:07.535964 10800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:07.543129 10800 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:07.543234 10800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:07.543357 10800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:12:07.551800 10800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:07.552973 10800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:07.554558 10800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:07.565793 10805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:07.566272 10806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:07.566715 10808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:07.567399 10800 server_base.cc:1034] running on GCE node
I20240501 08:12:07.567926 10800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:07.568800 10800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:07.569985 10800 hybrid_clock.cc:648] HybridClock initialized: now 1714551127569948 us; error 34 us; skew 500 ppm
I20240501 08:12:07.570283 10800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:07.572393 10800 webserver.cc:460] Webserver started at http://127.2.248.133:46771/ using document root <none> and password file <none>
I20240501 08:12:07.573086 10800 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:07.573210 10800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:07.573526 10800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:07.575882 10800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "bb2f0099fb794400b5c24927e213e5d8"
format_stamp: "Formatted at 2024-05-01 08:12:07 on dist-test-slave-0pdg"
I20240501 08:12:07.576591 10800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "bb2f0099fb794400b5c24927e213e5d8"
format_stamp: "Formatted at 2024-05-01 08:12:07 on dist-test-slave-0pdg"
I20240501 08:12:07.581279 10800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.002s
I20240501 08:12:07.584538 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:07.585808 10800 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240501 08:12:07.585935 10800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "bb2f0099fb794400b5c24927e213e5d8"
format_stamp: "Formatted at 2024-05-01 08:12:07 on dist-test-slave-0pdg"
I20240501 08:12:07.586098 10800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:07.608295 10800 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:07.609103 10800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:07.609318 10800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:07.610188 10800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:07.611552 10800 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:07.611632 10800 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:07.611711 10800 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:07.611763 10800 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:07.642678 10800 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:43981
I20240501 08:12:07.642731 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:43981 every 8 connection(s)
I20240501 08:12:07.643952 10800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:12:07.651511  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 10800
I20240501 08:12:07.651693  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:12:07.654743  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.134:0
--local_ip_for_outbound_sockets=127.2.248.134
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:07.656152 10927 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:07.656451 10927 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:07.657186 10927 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:07.658623 10221 ts_manager.cc:194] Registered new tserver with Master: bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981)
I20240501 08:12:07.659401 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:36521
I20240501 08:12:07.788401 10667 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
W20240501 08:12:07.969099 10930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:07.969406 10930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:07.969570 10930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:07.976781 10930 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:07.976893 10930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:07.977082 10930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.134
I20240501 08:12:07.985023 10930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.134
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:07.986166 10930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:07.988135 10930 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:08.000252 10935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:08.000778 10938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:08.001279 10936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:08.001973 10930 server_base.cc:1034] running on GCE node
I20240501 08:12:08.002518 10930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:08.003369 10930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:08.004640 10930 hybrid_clock.cc:648] HybridClock initialized: now 1714551128004581 us; error 43 us; skew 500 ppm
I20240501 08:12:08.004935 10930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:08.007143 10930 webserver.cc:460] Webserver started at http://127.2.248.134:39815/ using document root <none> and password file <none>
I20240501 08:12:08.007818 10930 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:08.007916 10930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:08.008318 10930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:08.013450 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "b6a964e54f3b44eda6d0666e0c1e4bea"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.014153 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "b6a964e54f3b44eda6d0666e0c1e4bea"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.019105 10930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240501 08:12:08.022475 10944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.023968 10930 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:12:08.024300 10930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b6a964e54f3b44eda6d0666e0c1e4bea"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.024588 10930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:08.041379 10930 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:08.042219 10930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:08.042469 10930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:08.043395 10930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:08.044714 10930 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:08.044804 10930 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.044905 10930 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:08.044971 10930 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.074455 10930 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.134:44821
I20240501 08:12:08.074512 11056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.134:44821 every 8 connection(s)
I20240501 08:12:08.075593 10930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:12:08.082653  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 10930
I20240501 08:12:08.082798  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:12:08.085933  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.135:0
--local_ip_for_outbound_sockets=127.2.248.135
--webserver_interface=127.2.248.135
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:08.087960 11057 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:08.088255 11057 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:08.089112 11057 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:08.090541 10221 ts_manager.cc:194] Registered new tserver with Master: b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
I20240501 08:12:08.091563 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.134:41129
I20240501 08:12:08.227532 10797 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
W20240501 08:12:08.419085 11060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:08.419338 11060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:08.419423 11060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:08.426504 11060 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:08.426617 11060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:08.426754 11060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.135
I20240501 08:12:08.435500 11060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.248.135
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.135
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:08.436728 11060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:08.438364 11060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:08.449824 11068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:08.450285 11066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:08.450814 11065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:08.451486 11060 server_base.cc:1034] running on GCE node
I20240501 08:12:08.452149 11060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:08.453220 11060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:08.454406 11060 hybrid_clock.cc:648] HybridClock initialized: now 1714551128454350 us; error 51 us; skew 500 ppm
I20240501 08:12:08.454659 11060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:08.456765 11060 webserver.cc:460] Webserver started at http://127.2.248.135:37863/ using document root <none> and password file <none>
I20240501 08:12:08.457401 11060 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:08.457501 11060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:08.457762 11060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:08.459909 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "441e281fafbf42f8938c3f2f42ff7527"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.460618 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "441e281fafbf42f8938c3f2f42ff7527"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.465440 11060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.002s
I20240501 08:12:08.468546 11074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.469700 11060 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.002s
I20240501 08:12:08.469835 11060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "441e281fafbf42f8938c3f2f42ff7527"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.469991 11060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:08.490271 11060 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:08.490921 11060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:08.491178 11060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:08.491992 11060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:08.493289 11060 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:08.493363 11060 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.493432 11060 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:08.493484 11060 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.523442 11060 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.135:37749
I20240501 08:12:08.523491 11186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.135:37749 every 8 connection(s)
I20240501 08:12:08.524838 11060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:12:08.532411  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 11060
I20240501 08:12:08.532565  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:12:08.535683  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.136:0
--local_ip_for_outbound_sockets=127.2.248.136
--webserver_interface=127.2.248.136
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:08.536819 11187 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:08.537128 11187 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:08.538105 11187 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:08.539482 10221 ts_manager.cc:194] Registered new tserver with Master: 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749)
I20240501 08:12:08.540525 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.135:42623
I20240501 08:12:08.661619 10927 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
W20240501 08:12:08.851188 11190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:08.851440 11190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:08.851529 11190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:08.858289 11190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:08.858379 11190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:08.858479 11190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.136
I20240501 08:12:08.866958 11190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.248.136
--webserver_port=0
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.136
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:08.868197 11190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:08.869963 11190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:08.888581 11195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:08.888551 11196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:08.890381 11198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:08.890990 11190 server_base.cc:1034] running on GCE node
I20240501 08:12:08.891819 11190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:08.893059 11190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:08.894327 11190 hybrid_clock.cc:648] HybridClock initialized: now 1714551128894234 us; error 73 us; skew 500 ppm
I20240501 08:12:08.894771 11190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:08.901803 11190 webserver.cc:460] Webserver started at http://127.2.248.136:34257/ using document root <none> and password file <none>
I20240501 08:12:08.902802 11190 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:08.902940 11190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:08.903410 11190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:08.907405 11190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c9f609481ded4d1aad9dd5f2341fd931"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.908669 11190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c9f609481ded4d1aad9dd5f2341fd931"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.915797 11190 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20240501 08:12:08.919281 11204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.920701 11190 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:08.920867 11190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c9f609481ded4d1aad9dd5f2341fd931"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:08.921036 11190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:08.938781 11190 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:08.939565 11190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:08.939812 11190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:08.940851 11190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:08.942293 11190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:08.942390 11190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.942485 11190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:08.942556 11190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:08.974496 11190 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.136:42153
I20240501 08:12:08.974547 11316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.136:42153 every 8 connection(s)
I20240501 08:12:08.976307 11190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:12:08.982932  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 11190
I20240501 08:12:08.983119  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:12:08.989823 11317 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:08.990083 11317 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:08.990965 11317 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:08.992480 10221 ts_manager.cc:194] Registered new tserver with Master: c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153)
I20240501 08:12:08.993196 10221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.136:58521
I20240501 08:12:09.000027  3042 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:12:09.018522  3042 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:12:09.018663  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10800
I20240501 08:12:09.027288 10922 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:09.094303 11057 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
I20240501 08:12:09.110301  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10800
I20240501 08:12:09.125685  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10930
I20240501 08:12:09.133890 11052 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:09.208737  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10930
I20240501 08:12:09.225077  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 11060
I20240501 08:12:09.233403 11182 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:09.310173  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 11060
I20240501 08:12:09.328524  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 11190
I20240501 08:12:09.337049 11312 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:12:09.418498  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 11190
I20240501 08:12:13.185138 10221 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:13.222278 10731 tablet_service.cc:1467] Processing CreateTablet for tablet 50b7150bbeb645d1b00d1a561c3276d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.222469 10602 tablet_service.cc:1467] Processing CreateTablet for tablet af24b8ab4d0c43ef986d89d4aefc263c (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.223304 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 84789f1228f64be5b6976a9d52f64bd0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.223505 10341 tablet_service.cc:1467] Processing CreateTablet for tablet 84789f1228f64be5b6976a9d52f64bd0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.224540 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84789f1228f64be5b6976a9d52f64bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.223625 10732 tablet_service.cc:1467] Processing CreateTablet for tablet af24b8ab4d0c43ef986d89d4aefc263c (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.224781 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84789f1228f64be5b6976a9d52f64bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.225356 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af24b8ab4d0c43ef986d89d4aefc263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.227742 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 50b7150bbeb645d1b00d1a561c3276d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.228354 10340 tablet_service.cc:1467] Processing CreateTablet for tablet 50b7150bbeb645d1b00d1a561c3276d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.228642 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 84789f1228f64be5b6976a9d52f64bd0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.228821 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b7150bbeb645d1b00d1a561c3276d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.229085 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b7150bbeb645d1b00d1a561c3276d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.229525 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84789f1228f64be5b6976a9d52f64bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.229676 10342 tablet_service.cc:1467] Processing CreateTablet for tablet af24b8ab4d0c43ef986d89d4aefc263c (DEFAULT_TABLE table=rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.230052 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af24b8ab4d0c43ef986d89d4aefc263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.230736 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b7150bbeb645d1b00d1a561c3276d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.237987 10602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af24b8ab4d0c43ef986d89d4aefc263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.246867 11346 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:13.250507 11346 tablet_bootstrap.cc:654] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.251823 11346 log.cc:824] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:13.257865 11349 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:13.262388 11349 tablet_bootstrap.cc:654] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.263337 11349 log.cc:824] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:13.265287 11351 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:13.265635 11346 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:13.265909 11346 ts_tablet_manager.cc:1367] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.002s
I20240501 08:12:13.266273 11347 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:13.268069 11349 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:13.268371 11349 ts_tablet_manager.cc:1367] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.007s
I20240501 08:12:13.269469 11351 tablet_bootstrap.cc:654] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.269587 11347 tablet_bootstrap.cc:654] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.271037 11351 log.cc:824] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:13.272418 11347 log.cc:824] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:13.276173 11346 raft_consensus.cc:348] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.276690 11346 raft_consensus.cc:374] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.276796 11346 raft_consensus.cc:729] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.277667 11346 consensus_queue.cc:260] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.278821 11349 raft_consensus.cc:348] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.279300 11349 raft_consensus.cc:374] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.279415 11349 raft_consensus.cc:729] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.280071 11349 consensus_queue.cc:260] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.280968 11351 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:13.281141 11346 ts_tablet_manager.cc:1398] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.006s
I20240501 08:12:13.281247 11351 ts_tablet_manager.cc:1367] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.006s
I20240501 08:12:13.281543 11346 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:13.283135 11346 tablet_bootstrap.cc:654] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.285729 11347 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:13.286000 11347 ts_tablet_manager.cc:1367] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.000s
I20240501 08:12:13.291795 11351 raft_consensus.cc:348] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.293308 11351 raft_consensus.cc:374] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.293422 11346 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:13.293526 11351 raft_consensus.cc:729] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.293586 11346 ts_tablet_manager.cc:1367] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.004s
I20240501 08:12:13.294312 11351 consensus_queue.cc:260] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.294662 11347 raft_consensus.cc:348] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.295084 11347 raft_consensus.cc:374] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.295226 11347 raft_consensus.cc:729] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.295794 11346 raft_consensus.cc:348] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.295897 11347 consensus_queue.cc:260] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.296041 11346 raft_consensus.cc:374] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.296167 11346 raft_consensus.cc:729] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.296454 11349 ts_tablet_manager.cc:1398] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.027s	user 0.013s	sys 0.011s
I20240501 08:12:13.296510 11346 consensus_queue.cc:260] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.297721 11349 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:13.297405 11346 ts_tablet_manager.cc:1398] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 08:12:13.299069 11346 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:13.299813 11349 tablet_bootstrap.cc:654] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.300700 11346 tablet_bootstrap.cc:654] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.301062 11347 ts_tablet_manager.cc:1398] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.008s
I20240501 08:12:13.301518 11347 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:13.303272 11347 tablet_bootstrap.cc:654] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.305941 11351 ts_tablet_manager.cc:1398] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.025s	user 0.015s	sys 0.000s
I20240501 08:12:13.306466 11351 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:13.306771 11347 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:13.306953 11347 ts_tablet_manager.cc:1367] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240501 08:12:13.308174 11347 raft_consensus.cc:348] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.308380 11347 raft_consensus.cc:374] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.308326 11351 tablet_bootstrap.cc:654] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.308482 11347 raft_consensus.cc:729] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.308780 11347 consensus_queue.cc:260] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.309453 11347 ts_tablet_manager.cc:1398] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240501 08:12:13.310830 11349 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:13.310981 11349 ts_tablet_manager.cc:1367] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.001s
I20240501 08:12:13.312023 11349 raft_consensus.cc:348] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.312886 11349 raft_consensus.cc:374] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.313014 11349 raft_consensus.cc:729] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.313256 11349 consensus_queue.cc:260] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.313506 11351 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:13.313678 11346 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:13.313859 11346 ts_tablet_manager.cc:1367] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.005s
I20240501 08:12:13.313911 11349 ts_tablet_manager.cc:1398] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:13.314193 11351 ts_tablet_manager.cc:1367] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240501 08:12:13.314561 11346 raft_consensus.cc:348] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.314767 11346 raft_consensus.cc:374] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.314846 11346 raft_consensus.cc:729] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.315078 11346 consensus_queue.cc:260] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.315168 11351 raft_consensus.cc:348] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.315388 11351 raft_consensus.cc:374] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.315481 11351 raft_consensus.cc:729] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.315591 11346 ts_tablet_manager.cc:1398] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:12:13.315675 11351 consensus_queue.cc:260] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.316190 11351 ts_tablet_manager.cc:1398] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
W20240501 08:12:13.333178 10538 tablet.cc:2369] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:13.372920 11357 raft_consensus.cc:482] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.373194 11357 raft_consensus.cc:504] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.375001 11357 leader_election.cc:288] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:13.382573 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84789f1228f64be5b6976a9d52f64bd0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:13.383051 10492 raft_consensus.cc:2454] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:13.383661 10555 leader_election.cc:302] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:13.383778 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84789f1228f64be5b6976a9d52f64bd0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:13.384013 11357 raft_consensus.cc:2781] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.384179 11357 raft_consensus.cc:482] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.384289 11357 raft_consensus.cc:3037] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.384362 10362 raft_consensus.cc:2454] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:13.386507 11357 raft_consensus.cc:504] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.387107 11357 leader_election.cc:288] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:13.387454 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84789f1228f64be5b6976a9d52f64bd0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:13.387647 10492 raft_consensus.cc:3037] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.387662 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84789f1228f64be5b6976a9d52f64bd0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:13.387853 10362 raft_consensus.cc:3037] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.389266 11358 raft_consensus.cc:482] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.389518 11358 raft_consensus.cc:504] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.390702 10492 raft_consensus.cc:2454] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:13.390810 10362 raft_consensus.cc:2454] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:13.391095 10555 leader_election.cc:302] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:13.391448 11358 leader_election.cc:288] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:13.392488 11357 raft_consensus.cc:2781] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
W20240501 08:12:13.393442 10408 tablet.cc:2369] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:13.400449 11357 raft_consensus.cc:686] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:13.400991 11357 consensus_queue.cc:237] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:13.410704 10220 catalog_manager.cc:5484] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:13.414456 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b7150bbeb645d1b00d1a561c3276d1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:13.414726 10362 raft_consensus.cc:2454] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:13.415544 10687 leader_election.cc:302] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:13.415988 11358 raft_consensus.cc:2781] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.416168 11358 raft_consensus.cc:482] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.416280 11358 raft_consensus.cc:3037] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.416245 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b7150bbeb645d1b00d1a561c3276d1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:13.416507 10492 raft_consensus.cc:2454] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:13.418438 11358 raft_consensus.cc:504] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.419031 11358 leader_election.cc:288] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:13.419451 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b7150bbeb645d1b00d1a561c3276d1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:13.419503 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b7150bbeb645d1b00d1a561c3276d1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:13.419672 10492 raft_consensus.cc:3037] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.419699 10362 raft_consensus.cc:3037] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.421531 10492 raft_consensus.cc:2454] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:13.421533 10362 raft_consensus.cc:2454] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:13.421934 10685 leader_election.cc:302] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 861f629e2d134ff08f25fa553dfb5bda; no voters: 
I20240501 08:12:13.422190 11358 raft_consensus.cc:2781] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:13.422616 11358 raft_consensus.cc:686] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Leader. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:13.423087 11358 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.427045 10220 catalog_manager.cc:5484] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1439c9ab8415485eff20566811ede (127.2.248.132). New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:13.463697 11358 raft_consensus.cc:482] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.463907 11358 raft_consensus.cc:504] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.464598 11358 leader_election.cc:288] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:13.465165 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af24b8ab4d0c43ef986d89d4aefc263c" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:13.465379 10362 raft_consensus.cc:2454] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
W20240501 08:12:13.465670 10798 tablet.cc:2369] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:13.465756 10687 leader_election.cc:302] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:13.466024 11358 raft_consensus.cc:2781] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.466128 11358 raft_consensus.cc:482] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.466238 11358 raft_consensus.cc:3037] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.468742 11358 raft_consensus.cc:504] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.469394 11358 leader_election.cc:288] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Requested vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:13.469920 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af24b8ab4d0c43ef986d89d4aefc263c" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:13.470108 10362 raft_consensus.cc:3037] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.471242 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af24b8ab4d0c43ef986d89d4aefc263c" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:13.471203 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af24b8ab4d0c43ef986d89d4aefc263c" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:13.471730 10622 raft_consensus.cc:2454] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:13.472400 10362 raft_consensus.cc:2454] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:13.472760 10687 leader_election.cc:302] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf
I20240501 08:12:13.473039 11358 raft_consensus.cc:2781] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:13.473218 11358 raft_consensus.cc:686] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Leader. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:13.473462 11358 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:13.475844 10220 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1439c9ab8415485eff20566811ede (127.2.248.132). New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:13.525334 10668 tablet.cc:2369] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:13.782938 11357 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.791301 11357 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:13.840755 11372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:13.841064 11372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:13.847971 11372 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:13.853956 11358 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.860585 11369 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 08:12:14.007853 10621 raft_consensus.cc:3037] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.016008 11386 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.025970 11386 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.088510 11413 mvcc.cc:205] Tried to move back new op lower bound from 7022801445005697024 to 7022801442709463040. Current Snapshot: MvccSnapshot[applied={T|T < 7022801445005697024}]

INSERT report
    rows total: 2000
    time total: 302.52 ms
  time per row: 0.15126 ms
I20240501 08:12:14.299777 10214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:14.308179 10472 tablet_service.cc:1467] Processing CreateTablet for tablet c665d7d20360440a9103e245f5f0900c (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.308619 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c665d7d20360440a9103e245f5f0900c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.309289 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 4b924c6344714a92ac37ae28961b02f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.309561 10602 tablet_service.cc:1467] Processing CreateTablet for tablet c665d7d20360440a9103e245f5f0900c (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.309734 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924c6344714a92ac37ae28961b02f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.309918 10600 tablet_service.cc:1467] Processing CreateTablet for tablet 850d4e1020e243f4b6fa113d2946846d (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.310345 10602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c665d7d20360440a9103e245f5f0900c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.310654 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d4e1020e243f4b6fa113d2946846d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.311029 10340 tablet_service.cc:1467] Processing CreateTablet for tablet 850d4e1020e243f4b6fa113d2946846d (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.311460 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d4e1020e243f4b6fa113d2946846d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.312980 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 4b924c6344714a92ac37ae28961b02f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.313387 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924c6344714a92ac37ae28961b02f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.316644 10731 tablet_service.cc:1467] Processing CreateTablet for tablet c665d7d20360440a9103e245f5f0900c (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.316707 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 4b924c6344714a92ac37ae28961b02f5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.317075 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924c6344714a92ac37ae28961b02f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.317077 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c665d7d20360440a9103e245f5f0900c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.321911 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 850d4e1020e243f4b6fa113d2946846d (DEFAULT_TABLE table=rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.322337 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d4e1020e243f4b6fa113d2946846d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.332774 11436 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:14.336452 11436 tablet_bootstrap.cc:654] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.339186 11435 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:14.339296 11434 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:14.340618 11437 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:14.341197 11435 tablet_bootstrap.cc:654] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.344563 11437 tablet_bootstrap.cc:654] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.344864 11436 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:14.345016 11436 ts_tablet_manager.cc:1367] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240501 08:12:14.346035 11434 tablet_bootstrap.cc:654] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.345925 11436 raft_consensus.cc:348] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.346170 11436 raft_consensus.cc:374] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.346268 11436 raft_consensus.cc:729] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.346601 11436 consensus_queue.cc:260] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.347268 11436 ts_tablet_manager.cc:1398] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:12:14.347577 11436 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:14.349184 11436 tablet_bootstrap.cc:654] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.352804 11435 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:14.353016 11435 ts_tablet_manager.cc:1367] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.007s
I20240501 08:12:14.353847 11435 raft_consensus.cc:348] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.354099 11435 raft_consensus.cc:374] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.354218 11435 raft_consensus.cc:729] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.354535 11435 consensus_queue.cc:260] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.355232 11435 ts_tablet_manager.cc:1398] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:12:14.357578 11435 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:14.359233 11435 tablet_bootstrap.cc:654] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.361234 11435 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:14.361409 11435 ts_tablet_manager.cc:1367] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.002s
I20240501 08:12:14.361974 11435 raft_consensus.cc:348] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.362130 11434 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:14.362143 11435 raft_consensus.cc:374] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.362226 11435 raft_consensus.cc:729] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.362327 11434 ts_tablet_manager.cc:1367] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.023s	user 0.002s	sys 0.005s
I20240501 08:12:14.362486 11435 consensus_queue.cc:260] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.362984 11435 ts_tablet_manager.cc:1398] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240501 08:12:14.363324 11434 raft_consensus.cc:348] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.363530 11434 raft_consensus.cc:374] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.363636 11434 raft_consensus.cc:729] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.364198 11436 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:14.364377 11436 ts_tablet_manager.cc:1367] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.013s
I20240501 08:12:14.364253 11434 consensus_queue.cc:260] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.364861 11437 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:14.365015 11437 ts_tablet_manager.cc:1367] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.027s	user 0.003s	sys 0.004s
I20240501 08:12:14.365833 11437 raft_consensus.cc:348] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.366044 11437 raft_consensus.cc:374] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.366153 11437 raft_consensus.cc:729] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.366377 11436 raft_consensus.cc:348] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.366408 11437 consensus_queue.cc:260] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.366681 11436 raft_consensus.cc:374] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.366797 11436 raft_consensus.cc:729] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.366973 11437 ts_tablet_manager.cc:1398] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:12:14.367151 11436 consensus_queue.cc:260] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.367681 11436 ts_tablet_manager.cc:1398] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:14.367058 11434 ts_tablet_manager.cc:1398] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.002s
I20240501 08:12:14.368093 11436 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:14.368146 11434 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:14.369727 11436 tablet_bootstrap.cc:654] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.369750 11434 tablet_bootstrap.cc:654] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.372905 11436 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:14.373039 11436 ts_tablet_manager.cc:1367] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20240501 08:12:14.373808 11434 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:14.373710 11436 raft_consensus.cc:348] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.373889 11436 raft_consensus.cc:374] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.373948 11434 ts_tablet_manager.cc:1367] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240501 08:12:14.373986 11436 raft_consensus.cc:729] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.374199 11436 consensus_queue.cc:260] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.374650 11436 ts_tablet_manager.cc:1398] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:12:14.374754 11434 raft_consensus.cc:348] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.374912 11434 raft_consensus.cc:374] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.374994 11434 raft_consensus.cc:729] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.375276 11434 consensus_queue.cc:260] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.376394 11434 ts_tablet_manager.cc:1398] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:12:14.376675 11434 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:14.378214 11434 tablet_bootstrap.cc:654] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.380066 11434 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:14.380226 11434 ts_tablet_manager.cc:1367] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.002s
I20240501 08:12:14.380743 11434 raft_consensus.cc:348] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.380887 11434 raft_consensus.cc:374] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.380980 11434 raft_consensus.cc:729] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.381178 11434 consensus_queue.cc:260] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.381603 11434 ts_tablet_manager.cc:1398] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:12:14.480458 11367 raft_consensus.cc:482] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.480638 11367 raft_consensus.cc:504] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.481384 11367 leader_election.cc:288] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:14.481978 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924c6344714a92ac37ae28961b02f5" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:14.482229 10492 raft_consensus.cc:2454] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:14.482751 10555 leader_election.cc:302] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:14.483040 11367 raft_consensus.cc:2781] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.483193 11367 raft_consensus.cc:482] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.483306 11367 raft_consensus.cc:3037] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.483680 11355 raft_consensus.cc:482] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.483868 11355 raft_consensus.cc:504] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.485337 11367 raft_consensus.cc:504] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.485638 11355 leader_election.cc:288] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:14.486013 11367 leader_election.cc:288] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:14.487018 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924c6344714a92ac37ae28961b02f5" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:14.487211 10492 raft_consensus.cc:3037] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.490357 10492 raft_consensus.cc:2454] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:14.490589 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924c6344714a92ac37ae28961b02f5" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:14.490842 10555 leader_election.cc:302] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:14.490818 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b924c6344714a92ac37ae28961b02f5" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:14.491041 10752 raft_consensus.cc:2454] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:14.491128 11367 raft_consensus.cc:2781] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.491279 11367 raft_consensus.cc:686] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:14.491575 11367 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.494139 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c665d7d20360440a9103e245f5f0900c" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:14.494351 10621 raft_consensus.cc:2454] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:14.494848 10425 leader_election.cc:302] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:14.495251 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c665d7d20360440a9103e245f5f0900c" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:14.495052 10213 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:14.495427 10751 raft_consensus.cc:2454] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:14.495659 11355 raft_consensus.cc:2781] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.495774 11355 raft_consensus.cc:482] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.495870 11355 raft_consensus.cc:3037] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.498163 11355 raft_consensus.cc:504] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.498713 11355 leader_election.cc:288] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Requested vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:14.499068 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c665d7d20360440a9103e245f5f0900c" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:14.499125 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c665d7d20360440a9103e245f5f0900c" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:14.499270 10621 raft_consensus.cc:3037] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.499291 10751 raft_consensus.cc:3037] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.502038 10751 raft_consensus.cc:2454] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:14.502053 10621 raft_consensus.cc:2454] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:14.502385 10425 leader_election.cc:302] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:14.502657 11355 raft_consensus.cc:2781] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.503049 11355 raft_consensus.cc:686] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Becoming Leader. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Running, Role: LEADER
I20240501 08:12:14.503511 11355 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:14.506546 10213 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130). New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:14.560474 11354 raft_consensus.cc:482] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.560672 11354 raft_consensus.cc:504] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.562495 11354 leader_election.cc:288] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:14.569272 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d4e1020e243f4b6fa113d2946846d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:14.569291 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d4e1020e243f4b6fa113d2946846d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:14.569540 10621 raft_consensus.cc:2454] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:14.569541 10492 raft_consensus.cc:2454] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:14.569947 10295 leader_election.cc:302] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:14.570255 11354 raft_consensus.cc:2781] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.570376 11354 raft_consensus.cc:482] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.570464 11354 raft_consensus.cc:3037] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.572353 11354 raft_consensus.cc:504] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.572813 11354 leader_election.cc:288] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:14.573297 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d4e1020e243f4b6fa113d2946846d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:14.573300 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d4e1020e243f4b6fa113d2946846d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:14.573495 10621 raft_consensus.cc:3037] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.573496 10492 raft_consensus.cc:3037] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.576265 10492 raft_consensus.cc:2454] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:14.576265 10621 raft_consensus.cc:2454] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:14.576687 10295 leader_election.cc:302] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:14.576999 11354 raft_consensus.cc:2781] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.577415 11354 raft_consensus.cc:686] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:14.577831 11354 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:14.581252 10213 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:14.902510 10751 raft_consensus.cc:3037] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.906299 11367 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.912570 11367 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:14.945974 11458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:14.946265 11458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:14.953518 11458 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:12:14.994277 11354 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.004096 11456 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.007856 11355 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.018707 11486 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 223.293 ms
  time per row: 0.111647 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:12:16.259330 10216 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:16.268465 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 92153666aebe47ddb080278cf24d1570 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.268520 10340 tablet_service.cc:1467] Processing CreateTablet for tablet 92153666aebe47ddb080278cf24d1570 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.268770 10731 tablet_service.cc:1467] Processing CreateTablet for tablet e99538651cf64510a811ba01c6c8fb40 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.268893 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92153666aebe47ddb080278cf24d1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.268904 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92153666aebe47ddb080278cf24d1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.269167 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99538651cf64510a811ba01c6c8fb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.272701 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 92153666aebe47ddb080278cf24d1570 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.273069 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92153666aebe47ddb080278cf24d1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.273034 10601 tablet_service.cc:1467] Processing CreateTablet for tablet e99538651cf64510a811ba01c6c8fb40 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.273475 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99538651cf64510a811ba01c6c8fb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.274267 10341 tablet_service.cc:1467] Processing CreateTablet for tablet e99538651cf64510a811ba01c6c8fb40 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.274340 10730 tablet_service.cc:1467] Processing CreateTablet for tablet 8f27eb42e75247c2aa142d3028e6ca2a (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.274610 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99538651cf64510a811ba01c6c8fb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.274699 10730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27eb42e75247c2aa142d3028e6ca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.280140 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 8f27eb42e75247c2aa142d3028e6ca2a (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.280370 10342 tablet_service.cc:1467] Processing CreateTablet for tablet 8f27eb42e75247c2aa142d3028e6ca2a (DEFAULT_TABLE table=rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.280552 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27eb42e75247c2aa142d3028e6ca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.280727 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27eb42e75247c2aa142d3028e6ca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.287248 11529 tablet_bootstrap.cc:492] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:16.290251 11529 tablet_bootstrap.cc:654] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.292855 11531 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:16.293227 11528 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:16.295372 11528 tablet_bootstrap.cc:654] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.295485 11531 tablet_bootstrap.cc:654] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.295941 11530 tablet_bootstrap.cc:492] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:16.297943 11531 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:16.298146 11531 ts_tablet_manager.cc:1367] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:12:16.298147 11530 tablet_bootstrap.cc:654] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.298921 11529 tablet_bootstrap.cc:492] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:16.298952 11531 raft_consensus.cc:348] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.299144 11529 ts_tablet_manager.cc:1367] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.007s
I20240501 08:12:16.299204 11531 raft_consensus.cc:374] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.299307 11531 raft_consensus.cc:729] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.299593 11531 consensus_queue.cc:260] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.300065 11528 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:16.300226 11528 ts_tablet_manager.cc:1367] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240501 08:12:16.300292 11530 tablet_bootstrap.cc:492] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:16.300316 11531 ts_tablet_manager.cc:1398] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:16.300417 11530 ts_tablet_manager.cc:1367] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20240501 08:12:16.300448 11529 raft_consensus.cc:348] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.300900 11531 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:16.300972 11528 raft_consensus.cc:348] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.301174 11528 raft_consensus.cc:374] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.301086 11530 raft_consensus.cc:348] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.301254 11528 raft_consensus.cc:729] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.300655 11529 raft_consensus.cc:374] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.301470 11529 raft_consensus.cc:729] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.301487 11528 consensus_queue.cc:260] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.301746 11529 consensus_queue.cc:260] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.302019 11528 ts_tablet_manager.cc:1398] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:12:16.302417 11531 tablet_bootstrap.cc:654] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.302512 11529 ts_tablet_manager.cc:1398] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:12:16.303081 11528 tablet_bootstrap.cc:492] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:16.307103 11528 tablet_bootstrap.cc:654] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.301288 11530 raft_consensus.cc:374] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.307845 11530 raft_consensus.cc:729] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.308228 11530 consensus_queue.cc:260] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.309593 11530 ts_tablet_manager.cc:1398] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240501 08:12:16.309851 11530 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:16.309937 11528 tablet_bootstrap.cc:492] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:16.310111 11528 ts_tablet_manager.cc:1367] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240501 08:12:16.310874 11528 raft_consensus.cc:348] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.311084 11528 raft_consensus.cc:374] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.311192 11528 raft_consensus.cc:729] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.311498 11530 tablet_bootstrap.cc:654] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.311419 11528 consensus_queue.cc:260] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.312907 11531 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:16.313083 11531 ts_tablet_manager.cc:1367] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.007s
I20240501 08:12:16.313854 11531 raft_consensus.cc:348] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.314085 11531 raft_consensus.cc:374] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.314208 11531 raft_consensus.cc:729] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.314517 11531 consensus_queue.cc:260] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.314846 11530 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:16.315120 11528 ts_tablet_manager.cc:1398] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.005s
I20240501 08:12:16.315258 11531 ts_tablet_manager.cc:1398] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:12:16.315557 11530 ts_tablet_manager.cc:1367] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240501 08:12:16.315599 11528 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:16.316516 11530 raft_consensus.cc:348] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.316789 11530 raft_consensus.cc:374] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.316893 11530 raft_consensus.cc:729] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.317169 11528 tablet_bootstrap.cc:654] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.317178 11530 consensus_queue.cc:260] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.317833 11530 ts_tablet_manager.cc:1398] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:16.318116 11530 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:16.319401 11530 tablet_bootstrap.cc:654] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.319772 11528 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:16.319911 11528 ts_tablet_manager.cc:1367] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 08:12:16.320608 11528 raft_consensus.cc:348] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.320860 11528 raft_consensus.cc:374] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.320998 11528 raft_consensus.cc:729] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.321178 11530 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:16.321297 11528 consensus_queue.cc:260] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.321359 11482 raft_consensus.cc:482] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.321360 11530 ts_tablet_manager.cc:1367] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.002s
I20240501 08:12:16.321520 11482 raft_consensus.cc:504] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.321990 11528 ts_tablet_manager.cc:1398] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:16.322350 11482 leader_election.cc:288] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:16.322587 11530 raft_consensus.cc:348] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.322866 11530 raft_consensus.cc:374] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.323004 11530 raft_consensus.cc:729] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.323086 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92153666aebe47ddb080278cf24d1570" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:16.323282 10492 raft_consensus.cc:2454] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:16.323300 11530 consensus_queue.cc:260] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.323948 11530 ts_tablet_manager.cc:1398] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:16.324296 10295 leader_election.cc:302] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:16.324553 11482 raft_consensus.cc:2781] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.324666 11482 raft_consensus.cc:482] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.324743 11482 raft_consensus.cc:3037] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.329910 11482 raft_consensus.cc:504] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.330718 11482 leader_election.cc:288] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:16.331058 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92153666aebe47ddb080278cf24d1570" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:16.331271 10492 raft_consensus.cc:3037] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.334182 10492 raft_consensus.cc:2454] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:16.334718 10295 leader_election.cc:302] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:16.334986 11482 raft_consensus.cc:2781] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.335320 11482 raft_consensus.cc:686] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:16.335608 11482 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:16.338496 10216 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:16.343860 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92153666aebe47ddb080278cf24d1570" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:16.343966 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92153666aebe47ddb080278cf24d1570" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:16.344091 10752 raft_consensus.cc:2454] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:16.401181 11482 raft_consensus.cc:482] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.401379 11482 raft_consensus.cc:504] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.402019 11482 leader_election.cc:288] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:16.402562 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99538651cf64510a811ba01c6c8fb40" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:16.402582 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99538651cf64510a811ba01c6c8fb40" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:16.402768 10622 raft_consensus.cc:2454] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:16.402997 10752 raft_consensus.cc:2454] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:16.403306 10295 leader_election.cc:302] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:16.403614 11482 raft_consensus.cc:2781] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.403704 11482 raft_consensus.cc:482] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.403811 11482 raft_consensus.cc:3037] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.405762 11482 raft_consensus.cc:504] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.406352 11482 leader_election.cc:288] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:16.406828 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99538651cf64510a811ba01c6c8fb40" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:16.407008 10752 raft_consensus.cc:3037] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.407189 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99538651cf64510a811ba01c6c8fb40" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:16.407354 10622 raft_consensus.cc:3037] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.409019 10752 raft_consensus.cc:2454] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:16.409207 10622 raft_consensus.cc:2454] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:16.409420 10297 leader_election.cc:302] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:16.409658 11482 raft_consensus.cc:2781] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.409817 11482 raft_consensus.cc:686] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:16.410084 11482 consensus_queue.cc:237] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.412694 10216 catalog_manager.cc:5484] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:16.579499 11509 raft_consensus.cc:482] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.579668 11509 raft_consensus.cc:504] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.580447 11509 leader_election.cc:288] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:16.580948 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27eb42e75247c2aa142d3028e6ca2a" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:16.581257 10752 raft_consensus.cc:2454] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:16.581661 10427 leader_election.cc:302] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 861f629e2d134ff08f25fa553dfb5bda; no voters: 
I20240501 08:12:16.581913 11509 raft_consensus.cc:2781] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.582029 11509 raft_consensus.cc:482] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.582161 11509 raft_consensus.cc:3037] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.584019 11509 raft_consensus.cc:504] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.584656 11509 leader_election.cc:288] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:16.585098 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27eb42e75247c2aa142d3028e6ca2a" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:16.585284 10752 raft_consensus.cc:3037] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.586735 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27eb42e75247c2aa142d3028e6ca2a" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:16.586841 10361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27eb42e75247c2aa142d3028e6ca2a" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:16.587005 10362 raft_consensus.cc:2454] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:16.587487 10752 raft_consensus.cc:2454] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:16.587790 10427 leader_election.cc:302] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 861f629e2d134ff08f25fa553dfb5bda; no voters: c3747ec2a93c4684962f215616e896ed
I20240501 08:12:16.588035 11509 raft_consensus.cc:2781] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.588444 11509 raft_consensus.cc:686] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Becoming Leader. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Running, Role: LEADER
I20240501 08:12:16.588721 11509 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:16.591759 10216 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130). New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:16.794500 10752 raft_consensus.cc:3037] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.797348 11482 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.803923 11544 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.882611 11555 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.891770 11482 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:16.993309 11551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:16.993592 11551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:17.000751 11551 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 08:12:17.097716 10361 raft_consensus.cc:3037] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.101189 11509 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.101797 11509 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:17.259519 10297 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 44469.4us

INSERT report
    rows total: 2000
    time total: 245.585 ms
  time per row: 0.122792 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:12:18.319185 10215 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:18.328320 10341 tablet_service.cc:1467] Processing CreateTablet for tablet 3453d8d883014219b1b14c88e250736b (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.328783 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3453d8d883014219b1b14c88e250736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.328749 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 3453d8d883014219b1b14c88e250736b (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.329165 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3453d8d883014219b1b14c88e250736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.331195 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 3453d8d883014219b1b14c88e250736b (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.331660 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3453d8d883014219b1b14c88e250736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.331777 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 4799a93d96a64a75b1984bb85fcad7e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.332226 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4799a93d96a64a75b1984bb85fcad7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.332392 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 4799a93d96a64a75b1984bb85fcad7e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.332691 10602 tablet_service.cc:1467] Processing CreateTablet for tablet 4799a93d96a64a75b1984bb85fcad7e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.333027 10602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4799a93d96a64a75b1984bb85fcad7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.336143 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4799a93d96a64a75b1984bb85fcad7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.336463 10600 tablet_service.cc:1467] Processing CreateTablet for tablet f5a58b7166534844b0963f4fab46dfd9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.331847 10731 tablet_service.cc:1467] Processing CreateTablet for tablet f5a58b7166534844b0963f4fab46dfd9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.336793 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a58b7166534844b0963f4fab46dfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.336911 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a58b7166534844b0963f4fab46dfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.340559 10342 tablet_service.cc:1467] Processing CreateTablet for tablet f5a58b7166534844b0963f4fab46dfd9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.340993 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a58b7166534844b0963f4fab46dfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.349655 11618 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:18.351537 11617 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:18.352025 11616 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:18.352567 11619 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:18.353251 11618 tablet_bootstrap.cc:654] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.353678 11617 tablet_bootstrap.cc:654] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.354247 11616 tablet_bootstrap.cc:654] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.357210 11619 tablet_bootstrap.cc:654] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.357295 11617 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:18.357437 11618 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:18.357497 11617 ts_tablet_manager.cc:1367] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240501 08:12:18.357590 11618 ts_tablet_manager.cc:1367] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240501 08:12:18.358330 11617 raft_consensus.cc:348] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.358337 11618 raft_consensus.cc:348] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.358661 11618 raft_consensus.cc:374] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.358723 11617 raft_consensus.cc:374] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.358762 11618 raft_consensus.cc:729] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.358825 11617 raft_consensus.cc:729] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.358987 11618 consensus_queue.cc:260] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.359164 11619 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:18.359050 11617 consensus_queue.cc:260] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.359473 11618 ts_tablet_manager.cc:1398] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:18.359591 11617 ts_tablet_manager.cc:1398] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:12:18.359776 11618 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:18.359875 11619 ts_tablet_manager.cc:1367] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240501 08:12:18.360662 11619 raft_consensus.cc:348] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.359907 11617 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:18.360921 11619 raft_consensus.cc:374] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.361021 11619 raft_consensus.cc:729] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.361287 11618 tablet_bootstrap.cc:654] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.361265 11619 consensus_queue.cc:260] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.362612 11619 ts_tablet_manager.cc:1398] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240501 08:12:18.362932 11619 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:18.363524 11618 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:18.363713 11618 ts_tablet_manager.cc:1367] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:12:18.364557 11618 raft_consensus.cc:348] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.364807 11618 raft_consensus.cc:374] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.364910 11618 raft_consensus.cc:729] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.365345 11618 consensus_queue.cc:260] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.365542 11619 tablet_bootstrap.cc:654] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.365933 11618 ts_tablet_manager.cc:1398] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:18.367153 11616 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:18.367394 11616 ts_tablet_manager.cc:1367] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I20240501 08:12:18.368500 11616 raft_consensus.cc:348] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.368790 11616 raft_consensus.cc:374] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.368937 11616 raft_consensus.cc:729] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.369221 11619 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:18.369169 11616 consensus_queue.cc:260] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.369372 11619 ts_tablet_manager.cc:1367] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.002s
I20240501 08:12:18.369562 11617 tablet_bootstrap.cc:654] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.370011 11619 raft_consensus.cc:348] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.370237 11619 raft_consensus.cc:374] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.370344 11619 raft_consensus.cc:729] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.370361 11616 ts_tablet_manager.cc:1398] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:18.370692 11616 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:18.370708 11619 consensus_queue.cc:260] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.372186 11619 ts_tablet_manager.cc:1398] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:18.373306 11616 tablet_bootstrap.cc:654] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.373397 11617 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:18.373561 11617 ts_tablet_manager.cc:1367] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.002s
I20240501 08:12:18.374367 11617 raft_consensus.cc:348] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.374563 11617 raft_consensus.cc:374] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.374636 11617 raft_consensus.cc:729] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.375198 11616 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:18.375144 11617 consensus_queue.cc:260] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.375505 11616 ts_tablet_manager.cc:1367] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:12:18.376259 11617 ts_tablet_manager.cc:1398] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:12:18.376230 11616 raft_consensus.cc:348] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.376441 11616 raft_consensus.cc:374] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.376536 11617 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:18.376518 11616 raft_consensus.cc:729] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.376760 11616 consensus_queue.cc:260] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.377779 11616 ts_tablet_manager.cc:1398] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:18.378449 11617 tablet_bootstrap.cc:654] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.380158 11617 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:18.380288 11617 ts_tablet_manager.cc:1367] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I20240501 08:12:18.380925 11617 raft_consensus.cc:348] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.381072 11617 raft_consensus.cc:374] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.381168 11617 raft_consensus.cc:729] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.381364 11617 consensus_queue.cc:260] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.381769 11617 ts_tablet_manager.cc:1398] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:12:18.417798 11587 raft_consensus.cc:482] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.417981 11587 raft_consensus.cc:504] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.418676 11587 leader_election.cc:288] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:18.419217 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4799a93d96a64a75b1984bb85fcad7e8" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:18.419215 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4799a93d96a64a75b1984bb85fcad7e8" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:18.419478 10752 raft_consensus.cc:2454] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:18.419478 10622 raft_consensus.cc:2454] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:18.419945 10425 leader_election.cc:302] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:18.420238 11587 raft_consensus.cc:2781] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.420368 11587 raft_consensus.cc:482] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.420485 11587 raft_consensus.cc:3037] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.422576 11587 raft_consensus.cc:504] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.423146 11587 leader_election.cc:288] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Requested vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:18.423535 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4799a93d96a64a75b1984bb85fcad7e8" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:18.423611 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4799a93d96a64a75b1984bb85fcad7e8" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:18.423705 10622 raft_consensus.cc:3037] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.423830 10752 raft_consensus.cc:3037] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.425441 10622 raft_consensus.cc:2454] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:18.425722 10752 raft_consensus.cc:2454] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:18.425802 10425 leader_election.cc:302] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:18.426100 11587 raft_consensus.cc:2781] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.426522 11587 raft_consensus.cc:686] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Becoming Leader. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Running, Role: LEADER
I20240501 08:12:18.426841 11587 consensus_queue.cc:237] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:18.429678 10215 catalog_manager.cc:5484] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130). New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:18.628221 11587 raft_consensus.cc:482] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.628036 11357 raft_consensus.cc:482] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.628476 11357 raft_consensus.cc:504] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.628473 11587 raft_consensus.cc:504] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.629168 11357 leader_election.cc:288] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:18.629220 11587 leader_election.cc:288] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:18.629781 10361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:18.629848 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:18.629837 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:18.629832 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:18.630044 10622 raft_consensus.cc:2454] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:18.630053 10492 raft_consensus.cc:2454] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:18.630040 10361 raft_consensus.cc:2454] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:18.630486 10555 leader_election.cc:302] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:18.630579 10425 leader_election.cc:302] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:18.630779 11357 raft_consensus.cc:2781] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.630885 11357 raft_consensus.cc:482] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.631006 11357 raft_consensus.cc:3037] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.631016 11587 raft_consensus.cc:2781] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.631121 11587 raft_consensus.cc:482] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.631243 11587 raft_consensus.cc:3037] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.633951 11357 raft_consensus.cc:504] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.633937 11587 raft_consensus.cc:504] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.634735 11587 leader_election.cc:288] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:18.634745 11357 leader_election.cc:288] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:18.635038 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:18.635022 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:18.635236 10362 raft_consensus.cc:3037] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.635191 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:18.635282 10622 raft_consensus.cc:2379] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 861f629e2d134ff08f25fa553dfb5bda in current term 1: Already voted for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in this term.
I20240501 08:12:18.635459 10492 raft_consensus.cc:2379] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in current term 1: Already voted for candidate 861f629e2d134ff08f25fa553dfb5bda in this term.
I20240501 08:12:18.636065 11555 raft_consensus.cc:482] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.636204 10361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453d8d883014219b1b14c88e250736b" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:18.636317 11555 raft_consensus.cc:504] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.636837 10557 leader_election.cc:302] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 861f629e2d134ff08f25fa553dfb5bda, c3747ec2a93c4684962f215616e896ed
I20240501 08:12:18.637058 11555 leader_election.cc:288] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:18.637349 11357 raft_consensus.cc:2726] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:18.637724 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a58b7166534844b0963f4fab46dfd9" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:18.637929 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a58b7166534844b0963f4fab46dfd9" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:18.638026 10752 raft_consensus.cc:2454] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:18.638150 10622 raft_consensus.cc:2454] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:18.638167 10362 raft_consensus.cc:2454] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:18.638509 10295 leader_election.cc:302] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:18.638552 10427 leader_election.cc:302] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, c3747ec2a93c4684962f215616e896ed; no voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf
I20240501 08:12:18.638816 11555 raft_consensus.cc:2781] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.638921 11555 raft_consensus.cc:482] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.638931 11587 raft_consensus.cc:2781] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.639029 11555 raft_consensus.cc:3037] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.639102 11587 raft_consensus.cc:686] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Becoming Leader. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Running, Role: LEADER
I20240501 08:12:18.639590 11587 consensus_queue.cc:237] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:18.641526 11555 raft_consensus.cc:504] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.642174 11555 leader_election.cc:288] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:18.642588 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a58b7166534844b0963f4fab46dfd9" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:18.642768 10622 raft_consensus.cc:3037] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.642709 10215 catalog_manager.cc:5484] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130). New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.645133 10622 raft_consensus.cc:2454] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:18.645622 10295 leader_election.cc:302] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:18.645897 11555 raft_consensus.cc:2781] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.646142 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a58b7166534844b0963f4fab46dfd9" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:18.646310 10752 raft_consensus.cc:3037] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.646636 11555 raft_consensus.cc:686] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:18.646898 11555 consensus_queue.cc:237] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:18.648519 10752 raft_consensus.cc:2454] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:18.649538 10215 catalog_manager.cc:5484] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:18.914932 11587 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.922137 11587 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:19.053323 11634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:19.053629 11634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:19.060864 11634 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:19.083973 11555 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.097965 11629 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.098834 11632 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
Using table 'rebalance_test_table_3'
I20240501 08:12:19.110699 11629 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 230.169 ms
  time per row: 0.115084 ms
I20240501 08:12:19.456880 10216 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:19.465793 10601 tablet_service.cc:1467] Processing CreateTablet for tablet bfbef7af4781448bb04b107b53d58ca0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.465960 10470 tablet_service.cc:1467] Processing CreateTablet for tablet bfbef7af4781448bb04b107b53d58ca0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.466284 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbef7af4781448bb04b107b53d58ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.466367 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbef7af4781448bb04b107b53d58ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.466334 10342 tablet_service.cc:1467] Processing CreateTablet for tablet bfbef7af4781448bb04b107b53d58ca0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.466722 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbef7af4781448bb04b107b53d58ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.468815 10731 tablet_service.cc:1467] Processing CreateTablet for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.469084 10341 tablet_service.cc:1467] Processing CreateTablet for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.469216 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.469486 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.470654 10732 tablet_service.cc:1467] Processing CreateTablet for tablet aa6139020ac7458aa8a7ac08f63e6f26 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.471051 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6139020ac7458aa8a7ac08f63e6f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.472786 10600 tablet_service.cc:1467] Processing CreateTablet for tablet aa6139020ac7458aa8a7ac08f63e6f26 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.473186 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6139020ac7458aa8a7ac08f63e6f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.476263 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.476649 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.480476 10471 tablet_service.cc:1467] Processing CreateTablet for tablet aa6139020ac7458aa8a7ac08f63e6f26 (DEFAULT_TABLE table=rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.480825 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6139020ac7458aa8a7ac08f63e6f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.487874 11698 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:19.489217 11696 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:19.490463 11698 tablet_bootstrap.cc:654] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.490449 11697 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:19.491019 11696 tablet_bootstrap.cc:654] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.492606 11697 tablet_bootstrap.cc:654] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.492591 11695 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:19.494802 11695 tablet_bootstrap.cc:654] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.494982 11696 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:19.495141 11696 ts_tablet_manager.cc:1367] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240501 08:12:19.495158 11698 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:19.495306 11698 ts_tablet_manager.cc:1367] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.005s
I20240501 08:12:19.495862 11696 raft_consensus.cc:348] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.496137 11696 raft_consensus.cc:374] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.496136 11698 raft_consensus.cc:348] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.496275 11696 raft_consensus.cc:729] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.496333 11698 raft_consensus.cc:374] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.496425 11698 raft_consensus.cc:729] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.496482 11697 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:19.496506 11696 consensus_queue.cc:260] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.496637 11697 ts_tablet_manager.cc:1367] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 08:12:19.496687 11698 consensus_queue.cc:260] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.496891 11695 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:19.497025 11696 ts_tablet_manager.cc:1398] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:12:19.497185 11698 ts_tablet_manager.cc:1398] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240501 08:12:19.497496 11698 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:19.497399 11697 raft_consensus.cc:348] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.497588 11697 raft_consensus.cc:374] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.497669 11697 raft_consensus.cc:729] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.497946 11696 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:19.497906 11697 consensus_queue.cc:260] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.498945 11698 tablet_bootstrap.cc:654] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.499446 11696 tablet_bootstrap.cc:654] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.499828 11695 ts_tablet_manager.cc:1367] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240501 08:12:19.500788 11697 ts_tablet_manager.cc:1398] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:12:19.502825 11695 raft_consensus.cc:348] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.504480 11695 raft_consensus.cc:374] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.504717 11695 raft_consensus.cc:729] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.505159 11695 consensus_queue.cc:260] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.506110 11698 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:19.506132 11695 ts_tablet_manager.cc:1398] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 08:12:19.506265 11698 ts_tablet_manager.cc:1367] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240501 08:12:19.506685 11695 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:19.506964 11698 raft_consensus.cc:348] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.507193 11698 raft_consensus.cc:374] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.507275 11698 raft_consensus.cc:729] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.507612 11698 consensus_queue.cc:260] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.508318 11698 ts_tablet_manager.cc:1398] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240501 08:12:19.508671 11698 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:19.509075 11695 tablet_bootstrap.cc:654] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.510103 11698 tablet_bootstrap.cc:654] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.511850 11697 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:19.512058 11698 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:19.512223 11698 ts_tablet_manager.cc:1367] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:12:19.512933 11698 raft_consensus.cc:348] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.513152 11698 raft_consensus.cc:374] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.513250 11698 raft_consensus.cc:729] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.513495 11698 consensus_queue.cc:260] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.514132 11698 ts_tablet_manager.cc:1398] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:19.515494 11697 tablet_bootstrap.cc:654] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.517030 11696 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:19.517196 11697 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:19.517253 11696 ts_tablet_manager.cc:1367] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.008s
I20240501 08:12:19.517424 11697 ts_tablet_manager.cc:1367] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.002s
I20240501 08:12:19.518060 11696 raft_consensus.cc:348] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.518141 11697 raft_consensus.cc:348] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.518358 11696 raft_consensus.cc:374] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.518399 11697 raft_consensus.cc:374] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.518484 11695 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:19.518481 11696 raft_consensus.cc:729] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.518520 11697 raft_consensus.cc:729] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.518657 11695 ts_tablet_manager.cc:1367] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.005s
I20240501 08:12:19.518744 11696 consensus_queue.cc:260] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.518882 11697 consensus_queue.cc:260] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.519459 11697 ts_tablet_manager.cc:1398] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:12:19.519908 11696 ts_tablet_manager.cc:1398] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:12:19.519999 11695 raft_consensus.cc:348] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.520241 11695 raft_consensus.cc:374] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.520318 11695 raft_consensus.cc:729] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.520551 11695 consensus_queue.cc:260] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.521351 11695 ts_tablet_manager.cc:1398] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:19.522327 11653 raft_consensus.cc:482] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.522491 11653 raft_consensus.cc:504] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.523278 11653 leader_election.cc:288] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:19.524045 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56f5f3cd9f4a56a3d43543ac0e5d14" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:19.524133 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56f5f3cd9f4a56a3d43543ac0e5d14" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:19.524277 10751 raft_consensus.cc:2454] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:19.524354 10492 raft_consensus.cc:2454] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:19.524736 10297 leader_election.cc:302] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:19.525250 11653 raft_consensus.cc:2781] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.525367 11653 raft_consensus.cc:482] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.525467 11653 raft_consensus.cc:3037] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.527307 11653 raft_consensus.cc:504] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.527809 11653 leader_election.cc:288] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:19.528244 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56f5f3cd9f4a56a3d43543ac0e5d14" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:19.528280 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e56f5f3cd9f4a56a3d43543ac0e5d14" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:19.528420 10492 raft_consensus.cc:3037] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.528489 10751 raft_consensus.cc:3037] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.530303 10751 raft_consensus.cc:2454] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:19.530334 10492 raft_consensus.cc:2454] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:19.530742 10295 leader_election.cc:302] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:19.531033 11653 raft_consensus.cc:2781] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.531208 11653 raft_consensus.cc:686] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:19.531498 11653 consensus_queue.cc:237] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.534246 10214 catalog_manager.cc:5484] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:19.663164 11629 raft_consensus.cc:482] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.663336 11629 raft_consensus.cc:504] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.663961 11629 leader_election.cc:288] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:19.664570 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbef7af4781448bb04b107b53d58ca0" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:19.664575 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbef7af4781448bb04b107b53d58ca0" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:19.664796 10362 raft_consensus.cc:2454] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:19.664801 10620 raft_consensus.cc:2454] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:19.665251 10427 leader_election.cc:302] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:19.665498 11629 raft_consensus.cc:2781] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.665601 11629 raft_consensus.cc:482] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.665692 11629 raft_consensus.cc:3037] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.667635 11629 raft_consensus.cc:504] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.668267 11629 leader_election.cc:288] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Requested vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:19.668732 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbef7af4781448bb04b107b53d58ca0" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:19.668733 10620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfbef7af4781448bb04b107b53d58ca0" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:19.668928 10362 raft_consensus.cc:3037] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.668910 10620 raft_consensus.cc:3037] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.670823 10620 raft_consensus.cc:2454] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:19.670970 10362 raft_consensus.cc:2454] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:19.671298 10425 leader_election.cc:302] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:19.671597 11629 raft_consensus.cc:2781] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.671766 11629 raft_consensus.cc:686] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Becoming Leader. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Running, Role: LEADER
I20240501 08:12:19.672080 11629 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:19.674832 10214 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130). New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:19.797039 11357 raft_consensus.cc:482] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.797260 11357 raft_consensus.cc:504] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.798022 11357 leader_election.cc:288] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:19.798599 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6139020ac7458aa8a7ac08f63e6f26" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:19.798620 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6139020ac7458aa8a7ac08f63e6f26" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:19.798831 10492 raft_consensus.cc:2454] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:19.798856 10751 raft_consensus.cc:2454] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:19.799280 10557 leader_election.cc:302] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:19.799539 11357 raft_consensus.cc:2781] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.799649 11357 raft_consensus.cc:482] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.799742 11357 raft_consensus.cc:3037] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.801649 11357 raft_consensus.cc:504] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.802173 11357 leader_election.cc:288] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:19.802639 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6139020ac7458aa8a7ac08f63e6f26" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:19.802641 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6139020ac7458aa8a7ac08f63e6f26" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:19.802822 10751 raft_consensus.cc:3037] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.802867 10492 raft_consensus.cc:3037] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.804791 10751 raft_consensus.cc:2454] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:19.805199 10492 raft_consensus.cc:2454] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:19.805202 10557 leader_election.cc:302] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:19.805512 11357 raft_consensus.cc:2781] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.805847 11357 raft_consensus.cc:686] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:19.806154 11357 consensus_queue.cc:237] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:19.808861 10214 catalog_manager.cc:5484] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:19.918185 11653 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.925791 11715 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.067431 11637 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.072103 11637 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:20.209069 11712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:20.209373 11712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:20.217016 11712 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:12:20.251755 11357 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.257252 11710 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:20.458035 10555 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.130:46605, user_credentials={real_user=slave}} blocked reactor thread for 53656us

INSERT report
    rows total: 2000
    time total: 237.749 ms
  time per row: 0.118874 ms
I20240501 08:12:20.607743 10216 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:20.616549 10341 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.617029 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.617203 10342 tablet_service.cc:1467] Processing CreateTablet for tablet cd01dd912e1e4ca9a104270f3cd004b1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.617532 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd01dd912e1e4ca9a104270f3cd004b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.617952 10731 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.618345 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.620504 10732 tablet_service.cc:1467] Processing CreateTablet for tablet cd01dd912e1e4ca9a104270f3cd004b1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.620839 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd01dd912e1e4ca9a104270f3cd004b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.621888 10340 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6f09b0b5e84329944360d511578ed0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.622257 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6f09b0b5e84329944360d511578ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.624557 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.624634 10600 tablet_service.cc:1467] Processing CreateTablet for tablet cd01dd912e1e4ca9a104270f3cd004b1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.624972 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.625591 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6f09b0b5e84329944360d511578ed0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.626103 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6f09b0b5e84329944360d511578ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.628633 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd01dd912e1e4ca9a104270f3cd004b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.628647 10602 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6f09b0b5e84329944360d511578ed0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.632664 10602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6f09b0b5e84329944360d511578ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.653831 11780 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:20.656833 11779 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:20.656993 11780 tablet_bootstrap.cc:654] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.661340 11779 tablet_bootstrap.cc:654] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.662058 11781 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:20.662655 11778 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:20.664572 11778 tablet_bootstrap.cc:654] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.664572 11781 tablet_bootstrap.cc:654] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.666810 11779 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:20.667042 11779 ts_tablet_manager.cc:1367] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.005s
I20240501 08:12:20.667464 11781 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:20.667637 11780 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:20.667646 11781 ts_tablet_manager.cc:1367] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240501 08:12:20.667862 11780 ts_tablet_manager.cc:1367] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.014s	user 0.000s	sys 0.011s
I20240501 08:12:20.668154 11779 raft_consensus.cc:348] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.668463 11779 raft_consensus.cc:374] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.668486 11781 raft_consensus.cc:348] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.668673 11781 raft_consensus.cc:374] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.667179 11778 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:20.668762 11781 raft_consensus.cc:729] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.668756 11780 raft_consensus.cc:348] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.668886 11778 ts_tablet_manager.cc:1367] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240501 08:12:20.668953 11780 raft_consensus.cc:374] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.669040 11780 raft_consensus.cc:729] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.669004 11781 consensus_queue.cc:260] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.669440 11780 consensus_queue.cc:260] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.669744 11778 raft_consensus.cc:348] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.669934 11778 raft_consensus.cc:374] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.670022 11778 raft_consensus.cc:729] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.670225 11780 ts_tablet_manager.cc:1398] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:12:20.670243 11778 consensus_queue.cc:260] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.670408 11781 ts_tablet_manager.cc:1398] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:20.670511 11780 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:20.670768 11778 ts_tablet_manager.cc:1398] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:12:20.671049 11778 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:20.672173 11780 tablet_bootstrap.cc:654] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.672673 11778 tablet_bootstrap.cc:654] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.676048 11779 raft_consensus.cc:729] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.678432 11780 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:20.678664 11780 ts_tablet_manager.cc:1367] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240501 08:12:20.676842 11779 consensus_queue.cc:260] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.679693 11780 raft_consensus.cc:348] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.679994 11780 raft_consensus.cc:374] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.680091 11780 raft_consensus.cc:729] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.680163 11779 ts_tablet_manager.cc:1398] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.001s
I20240501 08:12:20.680531 11779 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:20.680434 11780 consensus_queue.cc:260] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.681167 11780 ts_tablet_manager.cc:1398] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:12:20.681532 11780 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:20.683763 11780 tablet_bootstrap.cc:654] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.686288 11780 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:20.686471 11780 ts_tablet_manager.cc:1367] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240501 08:12:20.686734 11779 tablet_bootstrap.cc:654] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.687449 11780 raft_consensus.cc:348] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.687733 11780 raft_consensus.cc:374] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.687830 11780 raft_consensus.cc:729] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.688092 11780 consensus_queue.cc:260] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.688670 11779 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:20.688700 11780 ts_tablet_manager.cc:1398] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240501 08:12:20.688823 11779 ts_tablet_manager.cc:1367] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240501 08:12:20.688820 11778 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:20.688983 11778 ts_tablet_manager.cc:1367] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.009s
I20240501 08:12:20.689437 11779 raft_consensus.cc:348] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.689678 11779 raft_consensus.cc:374] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.689764 11779 raft_consensus.cc:729] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.689782 11778 raft_consensus.cc:348] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.690006 11778 raft_consensus.cc:374] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.690079 11778 raft_consensus.cc:729] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.689993 11779 consensus_queue.cc:260] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.690341 11778 consensus_queue.cc:260] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.690496 11779 ts_tablet_manager.cc:1398] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:12:20.690812 11778 ts_tablet_manager.cc:1398] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:12:20.691082 11778 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:20.692487 11778 tablet_bootstrap.cc:654] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.694442 11778 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:20.694561 11778 ts_tablet_manager.cc:1367] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:12:20.695204 11778 raft_consensus.cc:348] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.695421 11778 raft_consensus.cc:374] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.695533 11778 raft_consensus.cc:729] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.695755 11778 consensus_queue.cc:260] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.696382 11778 ts_tablet_manager.cc:1398] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:20.782212 11538 raft_consensus.cc:482] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.782430 11538 raft_consensus.cc:504] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.783211 11538 leader_election.cc:288] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:20.783715 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ed4ea6d6941d2af0f81eaf744bcea" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:20.783882 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ed4ea6d6941d2af0f81eaf744bcea" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:20.783946 10362 raft_consensus.cc:2454] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:20.784107 10621 raft_consensus.cc:2454] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:20.784437 10687 leader_election.cc:302] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:20.784780 11538 raft_consensus.cc:2781] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.784893 11538 raft_consensus.cc:482] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.784983 11538 raft_consensus.cc:3037] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.786855 11538 raft_consensus.cc:504] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.787415 11538 leader_election.cc:288] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:20.787863 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ed4ea6d6941d2af0f81eaf744bcea" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:20.788043 10362 raft_consensus.cc:3037] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.787997 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ed4ea6d6941d2af0f81eaf744bcea" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:20.788251 10621 raft_consensus.cc:3037] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.789929 10362 raft_consensus.cc:2454] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:20.790359 10687 leader_election.cc:302] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:20.790630 11538 raft_consensus.cc:2781] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.791088 11538 raft_consensus.cc:686] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Leader. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:20.791338 10621 raft_consensus.cc:2454] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:20.791517 11538 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:20.796893 10217 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1439c9ab8415485eff20566811ede (127.2.248.132). New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.832576 11762 raft_consensus.cc:482] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.832762 11762 raft_consensus.cc:504] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.833506 11762 leader_election.cc:288] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:20.833984 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6f09b0b5e84329944360d511578ed0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:20.833976 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6f09b0b5e84329944360d511578ed0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:20.834201 10491 raft_consensus.cc:2454] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:20.834237 10362 raft_consensus.cc:2454] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:20.834648 10557 leader_election.cc:302] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:20.834903 11762 raft_consensus.cc:2781] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.835003 11762 raft_consensus.cc:482] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.835068 11762 raft_consensus.cc:3037] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.837107 11762 raft_consensus.cc:504] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.837647 11762 leader_election.cc:288] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:20.838089 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6f09b0b5e84329944360d511578ed0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:20.838119 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6f09b0b5e84329944360d511578ed0" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:20.838282 10491 raft_consensus.cc:3037] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.838292 10362 raft_consensus.cc:3037] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.840134 10362 raft_consensus.cc:2454] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:20.840473 10491 raft_consensus.cc:2454] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:20.840553 10557 leader_election.cc:302] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:20.840873 11762 raft_consensus.cc:2781] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.841006 11762 raft_consensus.cc:686] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:20.841286 11762 consensus_queue.cc:237] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:20.844061 10217 catalog_manager.cc:5484] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.890087 11538 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.890347 11538 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.891151 11538 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:20.891723 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:20.891774 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:20.891989 10362 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:20.891999 10621 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:20.892488 10687 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:20.892809 11538 raft_consensus.cc:2781] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.892951 11538 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.893067 11538 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.895943 11538 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.896780 11538 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:20.897290 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:20.897540 10362 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.900583 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:20.900709 10362 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:20.900820 10621 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.901212 10687 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:20.901540 11538 raft_consensus.cc:2781] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.901751 11538 raft_consensus.cc:686] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Leader. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:20.902051 11538 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:20.903609 10621 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:20.905532 10217 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1439c9ab8415485eff20566811ede (127.2.248.132). New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:21.224210 11538 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.244323 11538 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:21.289994 11795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:21.290341 11795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:21.298723 11795 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:12:21.335788 11762 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.341154 11762 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.411944 11799 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.419561 11791 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 215.654 ms
  time per row: 0.107827 ms
I20240501 08:12:21.681929 10214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:21.690979 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 185a62881ea84131a0c14987e4296338 (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.691486 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185a62881ea84131a0c14987e4296338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.691428 10731 tablet_service.cc:1467] Processing CreateTablet for tablet 185a62881ea84131a0c14987e4296338 (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.691848 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185a62881ea84131a0c14987e4296338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.692592 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 6624441a1f1143648913595f905f8039 (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.692894 10340 tablet_service.cc:1467] Processing CreateTablet for tablet d188d415f63f448195ae90e8e3d66e0e (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.692975 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.693269 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188d415f63f448195ae90e8e3d66e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.696637 10342 tablet_service.cc:1467] Processing CreateTablet for tablet 185a62881ea84131a0c14987e4296338 (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.697034 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185a62881ea84131a0c14987e4296338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.698038 10730 tablet_service.cc:1467] Processing CreateTablet for tablet d188d415f63f448195ae90e8e3d66e0e (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.698427 10730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188d415f63f448195ae90e8e3d66e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.701839 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 6624441a1f1143648913595f905f8039 (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.701973 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 6624441a1f1143648913595f905f8039 (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.702370 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.702410 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.708170 10471 tablet_service.cc:1467] Processing CreateTablet for tablet d188d415f63f448195ae90e8e3d66e0e (DEFAULT_TABLE table=rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.708603 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188d415f63f448195ae90e8e3d66e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.722546 11863 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:21.723174 11864 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:21.723403 11861 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:21.724490 11863 tablet_bootstrap.cc:654] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.725188 11864 tablet_bootstrap.cc:654] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.725520 11862 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:21.725978 11861 tablet_bootstrap.cc:654] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.727542 11862 tablet_bootstrap.cc:654] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.733465 11864 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:21.733702 11864 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240501 08:12:21.733795 11863 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:21.734023 11863 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.007s
I20240501 08:12:21.734592 11864 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.734903 11864 raft_consensus.cc:374] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.734800 11863 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.734987 11863 raft_consensus.cc:374] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.735069 11864 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.735230 11863 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.735381 11864 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.735553 11863 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.735863 11862 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:21.736063 11862 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.010s
I20240501 08:12:21.736173 11864 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:12:21.736781 11863 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240501 08:12:21.737057 11862 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.738638 11863 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:21.740438 11863 tablet_bootstrap.cc:654] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.741963 11862 raft_consensus.cc:374] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.742290 11862 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.743016 11862 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.743921 11861 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:21.744212 11861 ts_tablet_manager.cc:1367] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.021s	user 0.006s	sys 0.005s
I20240501 08:12:21.744346 11862 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.002s
I20240501 08:12:21.744702 11863 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:21.744935 11863 ts_tablet_manager.cc:1367] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240501 08:12:21.745312 11862 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:21.745247 11861 raft_consensus.cc:348] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.745596 11861 raft_consensus.cc:374] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.745810 11863 raft_consensus.cc:348] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.746060 11863 raft_consensus.cc:374] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.746176 11863 raft_consensus.cc:729] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.746438 11863 consensus_queue.cc:260] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.746636 11861 raft_consensus.cc:729] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.746912 11861 consensus_queue.cc:260] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.747088 11863 ts_tablet_manager.cc:1398] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:21.747402 11862 tablet_bootstrap.cc:654] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.747495 11861 ts_tablet_manager.cc:1398] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:12:21.747637 11863 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:21.747804 11861 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:21.749584 11861 tablet_bootstrap.cc:654] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.751526 11863 tablet_bootstrap.cc:654] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.758608 11862 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:21.758818 11862 ts_tablet_manager.cc:1367] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.002s
I20240501 08:12:21.758862 11861 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:21.758903 11863 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:21.759017 11861 ts_tablet_manager.cc:1367] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240501 08:12:21.759073 11863 ts_tablet_manager.cc:1367] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.012s	user 0.009s	sys 0.000s
I20240501 08:12:21.759675 11863 raft_consensus.cc:348] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.759632 11862 raft_consensus.cc:348] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.759835 11863 raft_consensus.cc:374] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.759876 11862 raft_consensus.cc:374] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.759797 11861 raft_consensus.cc:348] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.759943 11863 raft_consensus.cc:729] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.759958 11862 raft_consensus.cc:729] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.760011 11861 raft_consensus.cc:374] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.760079 11861 raft_consensus.cc:729] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.760200 11863 consensus_queue.cc:260] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.760358 11861 consensus_queue.cc:260] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.760351 11862 consensus_queue.cc:260] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.761039 11861 ts_tablet_manager.cc:1398] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:21.761111 11862 ts_tablet_manager.cc:1398] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:12:21.761361 11862 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:21.763083 11862 tablet_bootstrap.cc:654] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.764880 11863 ts_tablet_manager.cc:1398] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 08:12:21.767199 11862 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:21.767323 11862 ts_tablet_manager.cc:1367] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.001s
I20240501 08:12:21.767983 11862 raft_consensus.cc:348] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.768175 11862 raft_consensus.cc:374] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.768278 11862 raft_consensus.cc:729] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.768493 11862 consensus_queue.cc:260] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.768925 11862 ts_tablet_manager.cc:1398] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20240501 08:12:21.844924 11869 raft_consensus.cc:482] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.845100 11869 raft_consensus.cc:504] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.845772 11869 leader_election.cc:288] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:21.846271 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185a62881ea84131a0c14987e4296338" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:21.846375 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185a62881ea84131a0c14987e4296338" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:21.846482 10751 raft_consensus.cc:2454] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:21.846634 10362 raft_consensus.cc:2454] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 0.
I20240501 08:12:21.846943 10427 leader_election.cc:302] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 861f629e2d134ff08f25fa553dfb5bda; no voters: 
I20240501 08:12:21.847218 11869 raft_consensus.cc:2781] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.847337 11869 raft_consensus.cc:482] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.847441 11869 raft_consensus.cc:3037] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.849339 11869 raft_consensus.cc:504] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.849897 11869 leader_election.cc:288] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:21.850394 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185a62881ea84131a0c14987e4296338" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:21.850422 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185a62881ea84131a0c14987e4296338" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:21.850621 10751 raft_consensus.cc:3037] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.850648 10362 raft_consensus.cc:3037] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.852665 10362 raft_consensus.cc:2454] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:21.852670 10751 raft_consensus.cc:2454] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 1.
I20240501 08:12:21.853078 10427 leader_election.cc:302] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 861f629e2d134ff08f25fa553dfb5bda; no voters: 
I20240501 08:12:21.853363 11869 raft_consensus.cc:2781] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.853526 11869 raft_consensus.cc:686] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Becoming Leader. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Running, Role: LEADER
I20240501 08:12:21.853792 11869 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.856838 10218 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130). New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.892516 11847 raft_consensus.cc:482] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.892732 11847 raft_consensus.cc:504] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.893491 11847 leader_election.cc:288] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:21.894042 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6624441a1f1143648913595f905f8039" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:21.894073 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6624441a1f1143648913595f905f8039" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:21.894335 10491 raft_consensus.cc:2454] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:21.894349 10751 raft_consensus.cc:2454] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:21.894832 10557 leader_election.cc:302] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:21.895120 11762 raft_consensus.cc:2781] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.895239 11762 raft_consensus.cc:482] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.895323 11762 raft_consensus.cc:3037] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.898255 11762 raft_consensus.cc:504] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.898939 11762 leader_election.cc:288] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:21.899437 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6624441a1f1143648913595f905f8039" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:21.899483 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6624441a1f1143648913595f905f8039" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:21.899689 10491 raft_consensus.cc:3037] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.899694 10751 raft_consensus.cc:3037] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.902618 10751 raft_consensus.cc:2454] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:21.902618 10491 raft_consensus.cc:2454] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:21.903148 10555 leader_election.cc:302] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:21.903545 11762 raft_consensus.cc:2781] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.903695 11762 raft_consensus.cc:686] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:21.903971 11762 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:21.907377 11555 raft_consensus.cc:482] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.907608 11555 raft_consensus.cc:504] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.908545 11555 leader_election.cc:288] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:21.908351 10218 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:21.909049 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188d415f63f448195ae90e8e3d66e0e" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:21.909284 10751 raft_consensus.cc:2454] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:21.909732 10297 leader_election.cc:302] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:21.909716 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188d415f63f448195ae90e8e3d66e0e" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:21.910030 10491 raft_consensus.cc:2454] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:21.910831 11555 raft_consensus.cc:2781] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.910951 11555 raft_consensus.cc:482] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.911083 11555 raft_consensus.cc:3037] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.913663 11555 raft_consensus.cc:504] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.914361 11555 leader_election.cc:288] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:21.914963 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188d415f63f448195ae90e8e3d66e0e" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:21.915163 10491 raft_consensus.cc:3037] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.915355 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d188d415f63f448195ae90e8e3d66e0e" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:21.915577 10751 raft_consensus.cc:3037] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.917222 10491 raft_consensus.cc:2454] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:21.918498 10751 raft_consensus.cc:2454] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:21.918987 10297 leader_election.cc:302] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:21.919335 11555 raft_consensus.cc:2781] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.919801 11555 raft_consensus.cc:686] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:21.920090 11555 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } }
I20240501 08:12:21.923074 10218 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } }
W20240501 08:12:22.319106 11879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:22.319444 11879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:12:22.323485 11555 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:22.328068 11879 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:22.331732 11762 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.346827 11555 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.367748 11869 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.368371 11847 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.392269 11897 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'

INSERT report
    rows total: 2000
    time total: 255.007 ms
  time per row: 0.127504 ms
I20240501 08:12:22.790356 10219 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:22.800668 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 03d0f89e1d8f40fe84d275b117d8145f (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.801133 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d0f89e1d8f40fe84d275b117d8145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.801056 10340 tablet_service.cc:1467] Processing CreateTablet for tablet 03d0f89e1d8f40fe84d275b117d8145f (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.801494 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d0f89e1d8f40fe84d275b117d8145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.801481 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 03d0f89e1d8f40fe84d275b117d8145f (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.801928 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d0f89e1d8f40fe84d275b117d8145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.807305 10602 tablet_service.cc:1467] Processing CreateTablet for tablet ea502ae00eff4f72a74b8d25f5a11ede (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.807693 10602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea502ae00eff4f72a74b8d25f5a11ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.807605 10472 tablet_service.cc:1467] Processing CreateTablet for tablet ea502ae00eff4f72a74b8d25f5a11ede (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.808096 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea502ae00eff4f72a74b8d25f5a11ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.808935 10342 tablet_service.cc:1467] Processing CreateTablet for tablet b72323b8b99f4fd1aedcce69089f6204 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.809305 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72323b8b99f4fd1aedcce69089f6204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.812918 10732 tablet_service.cc:1467] Processing CreateTablet for tablet ea502ae00eff4f72a74b8d25f5a11ede (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.813338 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea502ae00eff4f72a74b8d25f5a11ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.816023 10731 tablet_service.cc:1467] Processing CreateTablet for tablet b72323b8b99f4fd1aedcce69089f6204 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.816394 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72323b8b99f4fd1aedcce69089f6204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.816496 10600 tablet_service.cc:1467] Processing CreateTablet for tablet b72323b8b99f4fd1aedcce69089f6204 (DEFAULT_TABLE table=rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.816586 11946 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:22.816836 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72323b8b99f4fd1aedcce69089f6204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.818514 11946 tablet_bootstrap.cc:654] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.826820 11946 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:22.827019 11946 ts_tablet_manager.cc:1367] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.003s
I20240501 08:12:22.828313 11946 raft_consensus.cc:348] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.828672 11946 raft_consensus.cc:374] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.828796 11946 raft_consensus.cc:729] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.829349 11946 consensus_queue.cc:260] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.830149 11946 ts_tablet_manager.cc:1398] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:12:22.830628 11949 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:22.831733 11946 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:22.833693 11949 tablet_bootstrap.cc:654] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.833822 11946 tablet_bootstrap.cc:654] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.851243 11947 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:22.851660 11949 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:22.852460 11949 ts_tablet_manager.cc:1367] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.008s
I20240501 08:12:22.853837 11949 raft_consensus.cc:348] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.854645 11947 tablet_bootstrap.cc:654] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.854246 11949 raft_consensus.cc:374] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.855581 11949 raft_consensus.cc:729] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.856017 11950 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:22.855983 11949 consensus_queue.cc:260] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.856889 11949 ts_tablet_manager.cc:1398] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 08:12:22.857170 11949 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:22.858026 11946 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:22.858263 11946 ts_tablet_manager.cc:1367] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.012s
I20240501 08:12:22.858321 11950 tablet_bootstrap.cc:654] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.859256 11946 raft_consensus.cc:348] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.860827 11946 raft_consensus.cc:374] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.860978 11946 raft_consensus.cc:729] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.861066 11538 raft_consensus.cc:482] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.861130 11947 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:22.861274 11947 ts_tablet_manager.cc:1367] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240501 08:12:22.861215 11538 raft_consensus.cc:504] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.861574 11946 consensus_queue.cc:260] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.861868 11538 leader_election.cc:288] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:22.862474 11946 ts_tablet_manager.cc:1398] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 08:12:22.862493 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72323b8b99f4fd1aedcce69089f6204" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:22.862861 11947 raft_consensus.cc:348] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.863082 11947 raft_consensus.cc:374] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.864810 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72323b8b99f4fd1aedcce69089f6204" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:22.865034 10362 raft_consensus.cc:2454] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
W20240501 08:12:22.864977 10685 leader_election.cc:341] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:22.865473 10687 leader_election.cc:302] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf
I20240501 08:12:22.865734 11949 tablet_bootstrap.cc:654] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.865942 11538 raft_consensus.cc:2781] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.866081 11538 raft_consensus.cc:482] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.866194 11538 raft_consensus.cc:3037] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.867476 11950 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:22.867660 11950 ts_tablet_manager.cc:1367] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240501 08:12:22.868187 11949 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:22.868419 11949 ts_tablet_manager.cc:1367] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240501 08:12:22.868642 11950 raft_consensus.cc:348] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.868881 11950 raft_consensus.cc:374] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.868994 11950 raft_consensus.cc:729] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.869246 11950 consensus_queue.cc:260] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.869544 11949 raft_consensus.cc:348] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.869841 11949 raft_consensus.cc:374] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.869980 11949 raft_consensus.cc:729] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.870309 11949 consensus_queue.cc:260] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.871516 11950 ts_tablet_manager.cc:1398] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:12:22.871907 11950 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:22.870919 11538 raft_consensus.cc:504] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.874598 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72323b8b99f4fd1aedcce69089f6204" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:22.874820 10362 raft_consensus.cc:3037] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.875708 11950 tablet_bootstrap.cc:654] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.881107 10362 raft_consensus.cc:2454] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:22.881875 11538 leader_election.cc:288] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:22.882215 11947 raft_consensus.cc:729] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.882537 10687 leader_election.cc:302] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:22.882544 11947 consensus_queue.cc:260] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.882937 11538 raft_consensus.cc:2781] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.883117 11538 raft_consensus.cc:686] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Leader. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:22.883395 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72323b8b99f4fd1aedcce69089f6204" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:22.884405 11538 consensus_queue.cc:237] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
W20240501 08:12:22.883945 10685 leader_election.cc:341] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:22.888880 10216 catalog_manager.cc:5484] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1439c9ab8415485eff20566811ede (127.2.248.132). New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:22.892773 11950 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:22.893014 11950 ts_tablet_manager.cc:1367] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.021s	user 0.006s	sys 0.000s
I20240501 08:12:22.894196 11950 raft_consensus.cc:348] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.894515 11950 raft_consensus.cc:374] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.895308 11947 ts_tablet_manager.cc:1398] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.034s	user 0.002s	sys 0.000s
I20240501 08:12:22.895798 11947 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:22.894614 11950 raft_consensus.cc:729] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.901657 11950 consensus_queue.cc:260] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.902103 11947 tablet_bootstrap.cc:654] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.905058 11950 ts_tablet_manager.cc:1398] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.003s
I20240501 08:12:22.909374 11950 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:22.910961 11950 tablet_bootstrap.cc:654] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.911990 11947 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:22.912215 11947 ts_tablet_manager.cc:1367] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.007s
I20240501 08:12:22.912503 11949 ts_tablet_manager.cc:1398] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.044s	user 0.019s	sys 0.022s
I20240501 08:12:22.913096 11947 raft_consensus.cc:348] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.913352 11947 raft_consensus.cc:374] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.913655 11947 raft_consensus.cc:729] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.913992 11947 consensus_queue.cc:260] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.914793 11947 ts_tablet_manager.cc:1398] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:22.916585 11950 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:22.916757 11950 ts_tablet_manager.cc:1367] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.002s
I20240501 08:12:22.917508 11950 raft_consensus.cc:348] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.917703 11950 raft_consensus.cc:374] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.917788 11950 raft_consensus.cc:729] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.918000 11950 consensus_queue.cc:260] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.919200 11950 ts_tablet_manager.cc:1398] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:12:22.959995 11847 raft_consensus.cc:482] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.960228 11847 raft_consensus.cc:504] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.960888 11847 leader_election.cc:288] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:22.961467 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea502ae00eff4f72a74b8d25f5a11ede" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:22.961475 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea502ae00eff4f72a74b8d25f5a11ede" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:22.961681 10492 raft_consensus.cc:2454] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:22.961683 10751 raft_consensus.cc:2454] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:22.962095 10555 leader_election.cc:302] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:22.962435 11847 raft_consensus.cc:2781] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.962560 11847 raft_consensus.cc:482] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.962637 11847 raft_consensus.cc:3037] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.964666 11847 raft_consensus.cc:504] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.965212 11847 leader_election.cc:288] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:22.965653 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea502ae00eff4f72a74b8d25f5a11ede" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:22.965696 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea502ae00eff4f72a74b8d25f5a11ede" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:22.965864 10492 raft_consensus.cc:3037] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.965911 10751 raft_consensus.cc:3037] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.967998 10751 raft_consensus.cc:2454] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:22.968245 10492 raft_consensus.cc:2454] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:22.968436 10557 leader_election.cc:302] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:22.968760 11847 raft_consensus.cc:2781] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.968919 11847 raft_consensus.cc:686] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:22.969192 11847 consensus_queue.cc:237] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:22.971745 10216 catalog_manager.cc:5484] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:22.988070 11847 raft_consensus.cc:482] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.988348 11847 raft_consensus.cc:504] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.988940 11847 leader_election.cc:288] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:22.989492 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d0f89e1d8f40fe84d275b117d8145f" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:22.989516 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d0f89e1d8f40fe84d275b117d8145f" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:22.989708 10362 raft_consensus.cc:2454] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:22.989715 10492 raft_consensus.cc:2454] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:22.990159 10555 leader_election.cc:302] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:22.990443 11847 raft_consensus.cc:2781] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.990533 11847 raft_consensus.cc:482] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.990626 11847 raft_consensus.cc:3037] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.992823 11847 raft_consensus.cc:504] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.993441 11847 leader_election.cc:288] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:22.993881 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d0f89e1d8f40fe84d275b117d8145f" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:22.993908 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d0f89e1d8f40fe84d275b117d8145f" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:22.994073 10492 raft_consensus.cc:3037] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.994084 10362 raft_consensus.cc:3037] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.995843 10362 raft_consensus.cc:2454] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:22.995869 10492 raft_consensus.cc:2454] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:22.996312 10555 leader_election.cc:302] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:22.996654 11847 raft_consensus.cc:2781] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.996822 11847 raft_consensus.cc:686] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:22.997076 11847 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:22.999755 10216 catalog_manager.cc:5484] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:23.002296 11847 raft_consensus.cc:482] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.002460 11847 raft_consensus.cc:504] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:23.003126 11847 leader_election.cc:288] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:23.003507 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72323b8b99f4fd1aedcce69089f6204" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:23.003669 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72323b8b99f4fd1aedcce69089f6204" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:23.003783 10362 raft_consensus.cc:2379] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in current term 1: Already voted for candidate 06c1439c9ab8415485eff20566811ede in this term.
I20240501 08:12:23.004323 10557 leader_election.cc:302] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed
I20240501 08:12:23.004590 11847 raft_consensus.cc:3037] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.006840 11847 raft_consensus.cc:2726] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:23.361440 11956 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.370398 11956 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:23.408388 11963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:23.408733 11963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:23.416538 11963 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:23.432088 11762 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.438027 11847 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 08:12:23.549387 10492 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:23.550122 11762 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.550751 10362 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:23.551369 11762 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.609970 12012 mvcc.cc:205] Tried to move back new op lower bound from 7022801483970527232 to 7022801481718358016. Current Snapshot: MvccSnapshot[applied={T|T < 7022801483970527232}]
I20240501 08:12:23.650358 12008 mvcc.cc:205] Tried to move back new op lower bound from 7022801483970527232 to 7022801481718358016. Current Snapshot: MvccSnapshot[applied={T|T < 7022801483970527232}]
I20240501 08:12:23.666689 11969 mvcc.cc:205] Tried to move back new op lower bound from 7022801483970527232 to 7022801481718358016. Current Snapshot: MvccSnapshot[applied={T|T < 7022801483970527232 or (T in {7022801483970527232})}]

INSERT report
    rows total: 2000
    time total: 212.866 ms
  time per row: 0.106433 ms
I20240501 08:12:23.779194 10216 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:23.789017 10342 tablet_service.cc:1467] Processing CreateTablet for tablet bbef0147276d410da3d3830e682efe6d (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.789518 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbef0147276d410da3d3830e682efe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.790802 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 2a63e16f11894c2886091dcd3d204106 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.790849 10470 tablet_service.cc:1467] Processing CreateTablet for tablet bbef0147276d410da3d3830e682efe6d (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.790863 10471 tablet_service.cc:1467] Processing CreateTablet for tablet cf1f28d15c094fd98fac1b91d2785744 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.791333 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a63e16f11894c2886091dcd3d204106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.791532 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1f28d15c094fd98fac1b91d2785744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.791736 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbef0147276d410da3d3830e682efe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.792953 10601 tablet_service.cc:1467] Processing CreateTablet for tablet bbef0147276d410da3d3830e682efe6d (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.793381 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbef0147276d410da3d3830e682efe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.796540 10602 tablet_service.cc:1467] Processing CreateTablet for tablet cf1f28d15c094fd98fac1b91d2785744 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.796922 10602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1f28d15c094fd98fac1b91d2785744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.803673 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 2a63e16f11894c2886091dcd3d204106 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.803925 10340 tablet_service.cc:1467] Processing CreateTablet for tablet 2a63e16f11894c2886091dcd3d204106 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.804205 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a63e16f11894c2886091dcd3d204106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.804322 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a63e16f11894c2886091dcd3d204106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.803694 10731 tablet_service.cc:1467] Processing CreateTablet for tablet cf1f28d15c094fd98fac1b91d2785744 (DEFAULT_TABLE table=rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.809078 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1f28d15c094fd98fac1b91d2785744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.825390 12030 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:23.827399 12030 tablet_bootstrap.cc:654] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.829933 12031 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:23.832690 12031 tablet_bootstrap.cc:654] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.836527 12033 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:23.840684 12033 tablet_bootstrap.cc:654] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.843379 12029 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:23.843703 12030 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:23.844138 12031 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:23.843871 12030 ts_tablet_manager.cc:1367] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.019s	user 0.016s	sys 0.000s
I20240501 08:12:23.844441 12031 ts_tablet_manager.cc:1367] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.008s
I20240501 08:12:23.844928 12029 tablet_bootstrap.cc:654] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.844983 12030 raft_consensus.cc:348] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:23.845325 12030 raft_consensus.cc:374] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.845268 12031 raft_consensus.cc:348] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:23.845427 12030 raft_consensus.cc:729] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.845487 12031 raft_consensus.cc:374] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.845579 12031 raft_consensus.cc:729] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.845692 12030 consensus_queue.cc:260] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:23.845798 12031 consensus_queue.cc:260] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:23.846372 12031 ts_tablet_manager.cc:1398] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:12:23.846695 12031 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:23.847023 12033 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:23.847157 12033 ts_tablet_manager.cc:1367] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240501 08:12:23.847759 12033 raft_consensus.cc:348] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:23.848264 12031 tablet_bootstrap.cc:654] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.848300 12030 ts_tablet_manager.cc:1398] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:12:23.848639 12030 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:23.848021 12033 raft_consensus.cc:374] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.850387 12033 raft_consensus.cc:729] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.850522 12031 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:23.850716 12031 ts_tablet_manager.cc:1367] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:12:23.850651 12033 consensus_queue.cc:260] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:23.851527 12030 tablet_bootstrap.cc:654] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.851876 12031 raft_consensus.cc:348] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:23.852196 12031 raft_consensus.cc:374] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.852324 12031 raft_consensus.cc:729] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.852635 12031 consensus_queue.cc:260] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:23.853391 12033 ts_tablet_manager.cc:1398] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240501 08:12:23.853741 12033 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:23.855849 12031 ts_tablet_manager.cc:1398] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240501 08:12:23.856956 12033 tablet_bootstrap.cc:654] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.857265 12030 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:23.857415 12030 ts_tablet_manager.cc:1367] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240501 08:12:23.858119 12030 raft_consensus.cc:348] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:23.858438 12030 raft_consensus.cc:374] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.858521 12030 raft_consensus.cc:729] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.858796 12030 consensus_queue.cc:260] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:23.859004 12029 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:23.859102 12033 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:23.859262 12033 ts_tablet_manager.cc:1367] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240501 08:12:23.859377 12029 ts_tablet_manager.cc:1367] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.016s	user 0.000s	sys 0.004s
I20240501 08:12:23.860029 12030 ts_tablet_manager.cc:1398] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:23.860016 12033 raft_consensus.cc:348] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:23.860281 12033 raft_consensus.cc:374] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.860356 12033 raft_consensus.cc:729] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.860265 12029 raft_consensus.cc:348] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:23.860503 12029 raft_consensus.cc:374] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.860599 12029 raft_consensus.cc:729] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.860635 12033 consensus_queue.cc:260] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:23.860829 12029 consensus_queue.cc:260] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:23.861145 12033 ts_tablet_manager.cc:1398] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:23.861421 12029 ts_tablet_manager.cc:1398] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:12:23.861716 12029 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:23.863302 12029 tablet_bootstrap.cc:654] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.866611 12029 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:23.866782 12029 ts_tablet_manager.cc:1367] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240501 08:12:23.867496 12029 raft_consensus.cc:348] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:23.867694 12029 raft_consensus.cc:374] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.867765 12029 raft_consensus.cc:729] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.867986 12029 consensus_queue.cc:260] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:23.868563 12029 ts_tablet_manager.cc:1398] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:23.868875 12029 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:23.870530 12029 tablet_bootstrap.cc:654] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.872025 12029 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:23.872192 12029 ts_tablet_manager.cc:1367] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:12:23.872751 12029 raft_consensus.cc:348] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:23.872928 12029 raft_consensus.cc:374] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.873018 12029 raft_consensus.cc:729] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.873214 12029 consensus_queue.cc:260] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:23.873597 12029 ts_tablet_manager.cc:1398] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:12:24.047405 11956 raft_consensus.cc:482] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.047621 11956 raft_consensus.cc:504] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:24.048296 11956 leader_election.cc:288] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:24.048866 10361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a63e16f11894c2886091dcd3d204106" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:24.048919 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a63e16f11894c2886091dcd3d204106" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:24.049136 10361 raft_consensus.cc:2454] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:24.049193 10492 raft_consensus.cc:2454] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:24.049639 10687 leader_election.cc:302] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:24.049887 11956 raft_consensus.cc:2781] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.050001 11956 raft_consensus.cc:482] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.050084 11956 raft_consensus.cc:3037] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.052093 11956 raft_consensus.cc:504] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:24.052793 11956 leader_election.cc:288] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:24.053140 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a63e16f11894c2886091dcd3d204106" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:24.053349 10362 raft_consensus.cc:3037] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.053301 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a63e16f11894c2886091dcd3d204106" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:24.053506 10492 raft_consensus.cc:3037] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.056180 10362 raft_consensus.cc:2454] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:24.056223 10492 raft_consensus.cc:2454] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:24.056631 10685 leader_election.cc:302] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 861f629e2d134ff08f25fa553dfb5bda; no voters: 
I20240501 08:12:24.056937 11956 raft_consensus.cc:2781] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.057088 11956 raft_consensus.cc:686] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Leader. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:24.057381 11956 consensus_queue.cc:237] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:24.060256 10217 catalog_manager.cc:5484] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1439c9ab8415485eff20566811ede (127.2.248.132). New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:24.124567 11983 raft_consensus.cc:482] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.124758 11983 raft_consensus.cc:504] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:24.125366 11983 leader_election.cc:288] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:24.125758 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1f28d15c094fd98fac1b91d2785744" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:24.125883 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1f28d15c094fd98fac1b91d2785744" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:24.125981 10492 raft_consensus.cc:2454] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:24.126118 10751 raft_consensus.cc:2454] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:24.126382 10555 leader_election.cc:302] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:24.126675 11983 raft_consensus.cc:2781] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.126814 11983 raft_consensus.cc:482] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.126928 11983 raft_consensus.cc:3037] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.129462 11983 raft_consensus.cc:504] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:24.130267 11983 leader_election.cc:288] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:24.133539 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1f28d15c094fd98fac1b91d2785744" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:24.133786 10751 raft_consensus.cc:3037] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.133764 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1f28d15c094fd98fac1b91d2785744" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:24.134028 10492 raft_consensus.cc:3037] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.135973 10751 raft_consensus.cc:2454] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:24.136315 10492 raft_consensus.cc:2454] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:24.136453 10557 leader_election.cc:302] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:24.136816 11983 raft_consensus.cc:2781] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.136953 11983 raft_consensus.cc:686] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:24.137230 11983 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:24.140218 10217 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:24.199240 11983 raft_consensus.cc:482] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.199440 11983 raft_consensus.cc:504] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:24.199548 11887 raft_consensus.cc:482] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.199699 11887 raft_consensus.cc:504] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:24.200057 11983 leader_election.cc:288] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:24.200361 11887 leader_election.cc:288] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:24.200615 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:24.200639 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:24.200819 10362 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:24.200858 10492 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 0.
I20240501 08:12:24.200893 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:24.200958 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:24.201095 10491 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:24.201154 10619 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:24.201278 10555 leader_election.cc:302] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 
I20240501 08:12:24.201582 11983 raft_consensus.cc:2781] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.201591 10295 leader_election.cc:302] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:24.201664 11983 raft_consensus.cc:482] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.201733 11983 raft_consensus.cc:3037] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.201836 11887 raft_consensus.cc:2781] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.201932 11887 raft_consensus.cc:482] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.202034 11887 raft_consensus.cc:3037] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.204584 11983 raft_consensus.cc:504] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:24.204823 11887 raft_consensus.cc:504] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:24.205323 11983 leader_election.cc:288] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:24.205415 11887 leader_election.cc:288] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:24.205744 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:24.205890 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:24.205998 10491 raft_consensus.cc:3037] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.205957 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:24.206023 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:24.206175 10619 raft_consensus.cc:2379] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c3747ec2a93c4684962f215616e896ed in current term 1: Already voted for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in this term.
I20240501 08:12:24.206228 10362 raft_consensus.cc:2379] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in current term 1: Already voted for candidate c3747ec2a93c4684962f215616e896ed in this term.
I20240501 08:12:24.206656 10295 leader_election.cc:302] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3747ec2a93c4684962f215616e896ed; no voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf
I20240501 08:12:24.206893 11887 raft_consensus.cc:2726] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:24.207913 10491 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5f781e3ba94cbfbbea9b5782ef8eaf in term 1.
I20240501 08:12:24.208287 10555 leader_election.cc:302] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: c3747ec2a93c4684962f215616e896ed
I20240501 08:12:24.208622 11983 raft_consensus.cc:2781] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.208796 11983 raft_consensus.cc:686] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:24.209074 11983 consensus_queue.cc:237] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:24.212379 10217 catalog_manager.cc:5484] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:24.528553 11956 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.534312 11956 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:24.611672 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:24.612046 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:24.619896 12046 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 08:12:24.689901 12011 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.710147 12011 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.722628 12077 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.728336 12011 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 236.187 ms
  time per row: 0.118093 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:12:25.970692 10217 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51806:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:12:25.979673 10340 tablet_service.cc:1467] Processing CreateTablet for tablet 21065b7dcc18456eb1b4c2376d51eec4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.979854 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 21065b7dcc18456eb1b4c2376d51eec4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.980286 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21065b7dcc18456eb1b4c2376d51eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.980582 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 21065b7dcc18456eb1b4c2376d51eec4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.980895 10731 tablet_service.cc:1467] Processing CreateTablet for tablet ae946c165cd34853917e5c8a244b6408 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.981007 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21065b7dcc18456eb1b4c2376d51eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.981225 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae946c165cd34853917e5c8a244b6408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.983871 10341 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4ab8e42a484cf89b42f5f53f511265 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.984265 10342 tablet_service.cc:1467] Processing CreateTablet for tablet ae946c165cd34853917e5c8a244b6408 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.984440 10340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21065b7dcc18456eb1b4c2376d51eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.984642 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae946c165cd34853917e5c8a244b6408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.988073 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4ab8e42a484cf89b42f5f53f511265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.988018 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4ab8e42a484cf89b42f5f53f511265 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.988086 10602 tablet_service.cc:1467] Processing CreateTablet for tablet ae946c165cd34853917e5c8a244b6408 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.988444 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4ab8e42a484cf89b42f5f53f511265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.988543 10602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae946c165cd34853917e5c8a244b6408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.995759 10600 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4ab8e42a484cf89b42f5f53f511265 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.996363 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4ab8e42a484cf89b42f5f53f511265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.000592 12113 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:26.004475 12113 tablet_bootstrap.cc:654] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.005502 12114 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:26.008498 12115 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:12:26.010818 12115 tablet_bootstrap.cc:654] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.011931 12116 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:26.011941 12114 tablet_bootstrap.cc:654] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.012318 12113 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:26.012593 12113 ts_tablet_manager.cc:1367] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.005s
I20240501 08:12:26.013689 12113 raft_consensus.cc:348] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.014103 12113 raft_consensus.cc:374] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.014276 12113 raft_consensus.cc:729] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.014765 12116 tablet_bootstrap.cc:654] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.014801 12113 consensus_queue.cc:260] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.016963 12113 ts_tablet_manager.cc:1398] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 08:12:26.017392 12113 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:12:26.020727 12113 tablet_bootstrap.cc:654] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.023079 12115 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda: No bootstrap required, opened a new log
I20240501 08:12:26.023298 12115 ts_tablet_manager.cc:1367] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.010s
I20240501 08:12:26.023588 12113 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: No bootstrap required, opened a new log
I20240501 08:12:26.023624 12116 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:26.023756 12116 ts_tablet_manager.cc:1367] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240501 08:12:26.023761 12113 ts_tablet_manager.cc:1367] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.002s
I20240501 08:12:26.023864 12114 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:26.024065 12114 ts_tablet_manager.cc:1367] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.019s	user 0.001s	sys 0.006s
I20240501 08:12:26.024366 12115 raft_consensus.cc:348] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.024499 12113 raft_consensus.cc:348] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.024659 12115 raft_consensus.cc:374] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.024600 12116 raft_consensus.cc:348] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.024731 12113 raft_consensus.cc:374] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.024803 12116 raft_consensus.cc:374] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.024802 12113 raft_consensus.cc:729] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.024873 12115 raft_consensus.cc:729] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.024979 12116 raft_consensus.cc:729] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.025055 12113 consensus_queue.cc:260] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.025003 12114 raft_consensus.cc:348] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.025278 12114 raft_consensus.cc:374] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.025216 12116 consensus_queue.cc:260] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.025349 12115 consensus_queue.cc:260] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.025682 12113 ts_tablet_manager.cc:1398] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:26.025732 12116 ts_tablet_manager.cc:1398] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:26.026014 12116 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:26.026800 12115 ts_tablet_manager.cc:1398] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:26.027796 12114 raft_consensus.cc:729] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.028291 12114 consensus_queue.cc:260] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.030443 12114 ts_tablet_manager.cc:1398] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.001s
I20240501 08:12:26.030704 12114 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:26.032308 12116 tablet_bootstrap.cc:654] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.032334 12114 tablet_bootstrap.cc:654] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.034435 12114 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:26.034534 12116 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:26.034606 12114 ts_tablet_manager.cc:1367] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 08:12:26.034700 12116 ts_tablet_manager.cc:1367] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.003s
I20240501 08:12:26.035380 12114 raft_consensus.cc:348] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.035423 12116 raft_consensus.cc:348] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.035560 12114 raft_consensus.cc:374] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.035588 12116 raft_consensus.cc:374] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.035631 12114 raft_consensus.cc:729] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.035662 12116 raft_consensus.cc:729] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.035902 12116 consensus_queue.cc:260] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.035879 12114 consensus_queue.cc:260] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.036437 12114 ts_tablet_manager.cc:1398] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:26.036857 12114 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:12:26.037711 12116 ts_tablet_manager.cc:1398] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:12:26.038743 12116 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:26.039726 12114 tablet_bootstrap.cc:654] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.041210 12116 tablet_bootstrap.cc:654] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.043187 12114 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: No bootstrap required, opened a new log
I20240501 08:12:26.043206 12116 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: No bootstrap required, opened a new log
I20240501 08:12:26.043344 12116 ts_tablet_manager.cc:1367] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.001s
I20240501 08:12:26.043344 12114 ts_tablet_manager.cc:1367] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240501 08:12:26.044076 12116 raft_consensus.cc:348] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.044076 12114 raft_consensus.cc:348] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.044312 12116 raft_consensus.cc:374] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.044312 12114 raft_consensus.cc:374] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.044409 12116 raft_consensus.cc:729] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.044418 12114 raft_consensus.cc:729] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.044669 12114 consensus_queue.cc:260] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.044658 12116 consensus_queue.cc:260] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.045261 12116 ts_tablet_manager.cc:1398] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:26.045975 12114 ts_tablet_manager.cc:1398] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:12:26.079043 11967 raft_consensus.cc:482] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.079231 11967 raft_consensus.cc:504] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.079875 11967 leader_election.cc:288] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:26.080320 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21065b7dcc18456eb1b4c2376d51eec4" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:12:26.080523 10362 raft_consensus.cc:2454] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:26.080474 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21065b7dcc18456eb1b4c2376d51eec4" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:26.080761 10619 raft_consensus.cc:2454] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 0.
I20240501 08:12:26.080969 10687 leader_election.cc:302] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:26.081348 11967 raft_consensus.cc:2781] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.081454 11967 raft_consensus.cc:482] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.081571 11967 raft_consensus.cc:3037] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.083431 11967 raft_consensus.cc:504] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.084005 11967 leader_election.cc:288] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:26.084467 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21065b7dcc18456eb1b4c2376d51eec4" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:26.084690 10362 raft_consensus.cc:3037] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.084648 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21065b7dcc18456eb1b4c2376d51eec4" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:26.084834 10619 raft_consensus.cc:3037] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.086535 10362 raft_consensus.cc:2454] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:26.086732 10619 raft_consensus.cc:2454] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06c1439c9ab8415485eff20566811ede in term 1.
I20240501 08:12:26.086920 10687 leader_election.cc:302] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:26.087244 11967 raft_consensus.cc:2781] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.087658 11967 raft_consensus.cc:686] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Leader. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:26.087908 11967 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }
I20240501 08:12:26.090731 10214 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 06c1439c9ab8415485eff20566811ede (127.2.248.132). New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:26.299121 12108 raft_consensus.cc:482] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.299335 12108 raft_consensus.cc:504] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.300014 12108 leader_election.cc:288] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:26.300447 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:26.300505 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda" is_pre_election: true
I20240501 08:12:26.300659 10619 raft_consensus.cc:2454] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:26.300729 10492 raft_consensus.cc:2454] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:26.301002 10295 leader_election.cc:302] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:26.301226 12108 raft_consensus.cc:2781] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.301333 12108 raft_consensus.cc:482] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.301429 12108 raft_consensus.cc:3037] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.303268 12108 raft_consensus.cc:504] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.303740 12108 leader_election.cc:288] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:26.304210 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
I20240501 08:12:26.304217 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:26.304386 10492 raft_consensus.cc:3037] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.304455 10619 raft_consensus.cc:3037] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.306782 10492 raft_consensus.cc:2454] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:26.306874 10619 raft_consensus.cc:2454] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:26.307451 10295 leader_election.cc:302] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:26.307729 12108 raft_consensus.cc:2781] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.307852 12108 raft_consensus.cc:686] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:26.308130 12108 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:12:26.311349 10214 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:26.398813 12108 raft_consensus.cc:482] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.399001 12108 raft_consensus.cc:504] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.399590 12108 leader_election.cc:288] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:26.400079 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae946c165cd34853917e5c8a244b6408" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:12:26.400072 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae946c165cd34853917e5c8a244b6408" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:12:26.400291 10619 raft_consensus.cc:2454] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:26.400350 10752 raft_consensus.cc:2454] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 0.
I20240501 08:12:26.400681 10295 leader_election.cc:302] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:26.400959 12108 raft_consensus.cc:2781] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.401043 12108 raft_consensus.cc:482] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.401144 12108 raft_consensus.cc:3037] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.403048 12108 raft_consensus.cc:504] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.403622 12108 leader_election.cc:288] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:26.404059 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae946c165cd34853917e5c8a244b6408" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:26.404080 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae946c165cd34853917e5c8a244b6408" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:26.404315 10752 raft_consensus.cc:3037] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.404307 10619 raft_consensus.cc:3037] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.407123 10619 raft_consensus.cc:2454] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:26.407106 10752 raft_consensus.cc:2454] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 1.
I20240501 08:12:26.407507 10297 leader_election.cc:302] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:26.407757 12108 raft_consensus.cc:2781] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.407918 12108 raft_consensus.cc:686] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:26.408203 12108 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:12:26.411353 10214 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:26.593995 11967 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.600633 11967 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.723261 12108 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.729027 12138 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:26.806507 12131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:26.806835 12131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:26.814635 12131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 08:12:26.895346 12138 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.916594 12138 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 265.998 ms
  time per row: 0.132999 ms
I20240501 08:12:27.250036  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:43981
--local_ip_for_outbound_sockets=127.2.248.133
--tserver_master_addrs=127.2.248.190:33713
--webserver_port=46771
--webserver_interface=127.2.248.133
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:12:27.607440 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:27.607836 12191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:27.607980 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:27.616290 12191 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:27.616411 12191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:27.616557 12191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:12:27.625828 12191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:43981
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=46771
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:27.627180 12191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:27.628880 12191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:27.641249 12197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:27.641806 12200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:27.642400 12198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:27.642927 12191 server_base.cc:1034] running on GCE node
I20240501 08:12:27.643493 12191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:27.644454 12191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:27.645660 12191 hybrid_clock.cc:648] HybridClock initialized: now 1714551147645620 us; error 32 us; skew 500 ppm
I20240501 08:12:27.645978 12191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:27.648277 12191 webserver.cc:460] Webserver started at http://127.2.248.133:46771/ using document root <none> and password file <none>
I20240501 08:12:27.649020 12191 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:27.649149 12191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:27.654378 12191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.003s
I20240501 08:12:27.657080 12206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:27.658373 12191 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:12:27.658520 12191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "bb2f0099fb794400b5c24927e213e5d8"
format_stamp: "Formatted at 2024-05-01 08:12:07 on dist-test-slave-0pdg"
I20240501 08:12:27.659323 12191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:27.674325 12191 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:27.675140 12191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:27.675390 12191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:27.676481 12191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:27.678306 12191 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:27.678400 12191 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:27.678526 12191 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:27.678594 12191 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:27.712767 12191 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:43981
I20240501 08:12:27.712805 12318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:43981 every 8 connection(s)
I20240501 08:12:27.714149 12191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:12:27.719013  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 12191
I20240501 08:12:27.719844  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.134:44821
--local_ip_for_outbound_sockets=127.2.248.134
--tserver_master_addrs=127.2.248.190:33713
--webserver_port=39815
--webserver_interface=127.2.248.134
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:27.728930 12319 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:27.729243 12319 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:27.730046 12319 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:27.731535 10215 ts_manager.cc:194] Re-registered known tserver with Master: bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981)
I20240501 08:12:27.732635 10215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:44895
W20240501 08:12:28.076141 12323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:28.076491 12323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:28.076618 12323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:28.084443 12323 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:28.084590 12323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:28.084739 12323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.134
I20240501 08:12:28.093992 12323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.134:44821
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.248.134
--webserver_port=39815
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.134
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:28.095341 12323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:28.097079 12323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:28.109889 12331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:28.110535 12329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:28.110983 12328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:28.111734 12323 server_base.cc:1034] running on GCE node
I20240501 08:12:28.112406 12323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:28.113344 12323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:28.114557 12323 hybrid_clock.cc:648] HybridClock initialized: now 1714551148114491 us; error 53 us; skew 500 ppm
I20240501 08:12:28.114889 12323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:28.117241 12323 webserver.cc:460] Webserver started at http://127.2.248.134:39815/ using document root <none> and password file <none>
I20240501 08:12:28.117956 12323 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:28.118086 12323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:28.123415 12323 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 08:12:28.126075 12337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:28.127466 12323 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:12:28.127636 12323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b6a964e54f3b44eda6d0666e0c1e4bea"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:28.128643 12323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:28.143550 12323 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:28.144408 12323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:28.144675 12323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:28.145921 12323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:28.147672 12323 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:28.147748 12323 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:28.147917 12323 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:28.147969 12323 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:28.188922 12323 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.134:44821
I20240501 08:12:28.188959 12449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.134:44821 every 8 connection(s)
I20240501 08:12:28.190418 12323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:12:28.190958  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 12323
I20240501 08:12:28.191675  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.135:37749
--local_ip_for_outbound_sockets=127.2.248.135
--tserver_master_addrs=127.2.248.190:33713
--webserver_port=37863
--webserver_interface=127.2.248.135
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:28.204556 12450 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:28.204893 12450 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:28.205879 12450 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:28.207463 10215 ts_manager.cc:194] Re-registered known tserver with Master: b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
I20240501 08:12:28.208321 10215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.134:51765
W20240501 08:12:28.544256 12453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:28.544531 12453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:28.544629 12453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:28.552469 12453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:28.552588 12453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:28.552716 12453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.135
I20240501 08:12:28.561848 12453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.135:37749
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.2.248.135
--webserver_port=37863
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.135
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:28.563338 12453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:28.565104 12453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:28.577610 12459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:28.578025 12462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:28.578615 12460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:28.579336 12453 server_base.cc:1034] running on GCE node
I20240501 08:12:28.579949 12453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:28.581130 12453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:28.582351 12453 hybrid_clock.cc:648] HybridClock initialized: now 1714551148582307 us; error 53 us; skew 500 ppm
I20240501 08:12:28.582734 12453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:28.585371 12453 webserver.cc:460] Webserver started at http://127.2.248.135:37863/ using document root <none> and password file <none>
I20240501 08:12:28.586133 12453 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:28.586243 12453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:28.591533 12453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240501 08:12:28.594725 12468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:28.596282 12453 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:28.596443 12453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "441e281fafbf42f8938c3f2f42ff7527"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:28.597340 12453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:28.614565 12453 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:28.615389 12453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:28.615675 12453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:28.616642 12453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:28.618127 12453 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:28.618203 12453 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:28.618322 12453 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:28.618374 12453 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:28.654400 12453 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.135:37749
I20240501 08:12:28.654458 12580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.135:37749 every 8 connection(s)
I20240501 08:12:28.655835 12453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:12:28.663730  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 12453
I20240501 08:12:28.664438  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.136:42153
--local_ip_for_outbound_sockets=127.2.248.136
--tserver_master_addrs=127.2.248.190:33713
--webserver_port=34257
--webserver_interface=127.2.248.136
--builtin_ntp_servers=127.2.248.148:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:28.668321 12581 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:28.668607 12581 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:28.669546 12581 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:28.670979 10215 ts_manager.cc:194] Re-registered known tserver with Master: 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749)
I20240501 08:12:28.672199 10215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.135:34411
I20240501 08:12:28.736088 12319 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
W20240501 08:12:29.012351 12584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:29.012655 12584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:29.012777 12584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:29.020495 12584 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:29.020628 12584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:29.020792 12584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.136
I20240501 08:12:29.031476 12584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.136:42153
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.2.248.136
--webserver_port=34257
--tserver_master_addrs=127.2.248.190:33713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.136
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:29.032920 12584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:29.034844 12584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:29.047784 12589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:29.048183 12592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:29.048693 12590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:29.049438 12584 server_base.cc:1034] running on GCE node
I20240501 08:12:29.050009 12584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:29.050892 12584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:29.052104 12584 hybrid_clock.cc:648] HybridClock initialized: now 1714551149052017 us; error 61 us; skew 500 ppm
I20240501 08:12:29.052459 12584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:29.054664 12584 webserver.cc:460] Webserver started at http://127.2.248.136:34257/ using document root <none> and password file <none>
I20240501 08:12:29.055387 12584 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:29.055508 12584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:29.060624 12584 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:29.063226 12599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:29.064637 12584 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:29.064790 12584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c9f609481ded4d1aad9dd5f2341fd931"
format_stamp: "Formatted at 2024-05-01 08:12:08 on dist-test-slave-0pdg"
I20240501 08:12:29.065593 12584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:29.089711 12584 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:29.090533 12584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:29.090803 12584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:29.091845 12584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:29.093577 12584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:29.093669 12584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:29.093803 12584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:29.093861 12584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:29.135843 12584 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.136:42153
I20240501 08:12:29.135879 12711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.136:42153 every 8 connection(s)
I20240501 08:12:29.137547 12584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:12:29.143797  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 12584
I20240501 08:12:29.154021 12712 heartbeater.cc:347] Connected to a master server at 127.2.248.190:33713
I20240501 08:12:29.154315 12712 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:29.155189 12712 heartbeater.cc:511] Master 127.2.248.190:33713 requested a full tablet report, sending...
I20240501 08:12:29.156761 10220 ts_manager.cc:194] Re-registered known tserver with Master: c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153)
I20240501 08:12:29.157948 10220 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.136:36977
I20240501 08:12:29.211136 12450 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
I20240501 08:12:29.674963 12581 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
I20240501 08:12:30.162678 12712 heartbeater.cc:503] Master 127.2.248.190:33713 was elected leader, sending a full tablet report...
I20240501 08:12:30.385970 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:30.387123 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:30.464495 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:30.475867 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.497637 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:30.498200 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:30.514596 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.574221 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:30.576402 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.610368 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.611114 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.635609 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.653198 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:30.713661 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.763613 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.794147 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:30.989455 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.994547 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:30.997751 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:31.048552 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:31.079210 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.112855 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.115135 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:31.125367 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.309535 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:31.348485 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.393108 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:31.396277 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.413282 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.431444 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.450341 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.455152 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:31.462497 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:31.481796 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:31.488581 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:31.508821 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.516052 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:31.571110 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:31.653594 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:31.674010 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.265340 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:32.301586 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:32.310279 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:32.339427 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.339478 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.349624 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.373805 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.378266 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:32.392306 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.397323 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:32.397723 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.413614 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:32.461012 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:32.511052 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.516362 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:32.523597 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:32.548005 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.566387 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.613785 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:32.621482 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:32.639914 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.641830 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.674782 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:32.680502 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.951062 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:32.976900 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:32.997263 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.998142 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.998946 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:32.999878 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:33.051065 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:33.060096 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.066334 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:33.073017 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:33.101898 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.150382 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.161432 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:33.167938 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.273344 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.294754 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:33.859501 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:33.867628 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.876750 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.882292 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:33.909626 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.993851 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.998919 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:34.033506 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:34.182839 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.188452 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:34.189339 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:34.207652 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:34.243953 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.264595 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:34.266819 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:34.291407 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.317831 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:34.321178 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.337747 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:34.349184 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.351908 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.373190 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.420464 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:34.470582 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.680972 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:34.687397 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.703297 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.718031 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:34.719972 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:34.721091 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:34.728020 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.761057 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:34.804495 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.814910 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:34.828269 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.812131 10491 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:34.841163 10751 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:34.844035 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.848809 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:34.849428 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:34.880057 10752 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:34.880154 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:34.883404 13322 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:34.884928 10362 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:34.889057 13001 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:34.890215 13322 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:34.892555 10752 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:34.897678 10215 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:34.931053 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:12:34.957769 10425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.131:34319, user_credentials={real_user=slave}} blocked reactor thread for 62790.9us
W20240501 08:12:34.980528 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 80143.9us
I20240501 08:12:34.981280 10362 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:34.994432 10427 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f27eb42e75247c2aa142d3028e6ca2a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.000982 10620 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.001761 13405 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.015256 10362 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:35.027904 10361 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.028517 13405 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.037827 10491 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.038179 10752 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.039024 12873 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.039852 12873 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:35.054916 10295 consensus_peers.cc:485] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d188d415f63f448195ae90e8e3d66e0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.064416 10619 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:35.072502 13497 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.074769 10490 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.078570 10620 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.069773 13405 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.081072 10220 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:35.088169 10297 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 49129.1us
W20240501 08:12:35.093946 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af24b8ab4d0c43ef986d89d4aefc263c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.096680 10220 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.097627 10362 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.127637 10752 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.133394 10491 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.134220 12992 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.134457 10751 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.136255 12992 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.156842 12992 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.161150 10490 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.166428 10491 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:35.167980 10221 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:12:35.177480 10555 consensus_peers.cc:485] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf1f28d15c094fd98fac1b91d2785744. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.178054 10361 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:35.184509 10752 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:12:35.191146 10427 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c665d7d20360440a9103e245f5f0900c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.193267 10619 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.193326 10752 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.193827 13473 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.194115 13477 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.198493 13473 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.202347 10751 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.209592 10220 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.241626 10751 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.242694 12873 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:35.245311 10297 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae946c165cd34853917e5c8a244b6408. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.246001 10620 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.246711 12873 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.274106 13497 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.279996 10752 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.277602 10620 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.286355 10215 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.300582 10619 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.326699 10751 consensus_queue.cc:237] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:35.330964 10361 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.331511 13405 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.335680 10619 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.336722 13405 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.347204 10619 consensus_queue.cc:237] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:35.358107 13405 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.363471 10362 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.364075 10221 catalog_manager.cc:5484] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.373195 10491 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:35.374789 10621 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.376993 10361 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.378362 10362 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.379038 10751 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.379875 13513 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:35.379997 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b72323b8b99f4fd1aedcce69089f6204. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:35.381667 10427 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f27eb42e75247c2aa142d3028e6ca2a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.382237 13473 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.382686 13473 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.385995 13001 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.388872 10751 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:12:35.390465 10425 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f27eb42e75247c2aa142d3028e6ca2a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.395089 10360 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.401506 10221 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.409528 13540 ts_tablet_manager.cc:902] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:35.409740 10491 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.410708 13513 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:35.414966 10558 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbef0147276d410da3d3830e682efe6d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.425557 12992 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.429255 10360 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.433990 10220 catalog_manager.cc:5484] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.438077 10491 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.456969 13558 ts_tablet_manager.cc:902] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:35.469002 10490 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:35.473366 10619 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.473567 10360 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:35.474292 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfbef7af4781448bb04b107b53d58ca0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.474519 13473 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.475687 13001 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.489745 10620 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:35.491586 13540 tablet_copy_client.cc:308] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:35.493849 13001 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.497262 10360 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.502717 10215 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:35.506304 10558 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6624441a1f1143648913595f905f8039. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.507561 10751 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.508183 13382 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.520895 13558 tablet_copy_client.cc:308] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:35.522553 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 8f27eb42e75247c2aa142d3028e6ca2a from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:37801)
I20240501 08:12:35.522817 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 8f27eb42e75247c2aa142d3028e6ca2a from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:37801: session id = c9f609481ded4d1aad9dd5f2341fd931-8f27eb42e75247c2aa142d3028e6ca2a
I20240501 08:12:35.522583 13382 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.527971 10512 tablet_copy_source_session.cc:205] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:35.528221 10490 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.528766 10751 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.529680 10220 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.538542 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27eb42e75247c2aa142d3028e6ca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.541843 13513 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.547286 10360 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:35.556247 10295 consensus_peers.cc:485] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d188d415f63f448195ae90e8e3d66e0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.558338 10752 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.559123 12873 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.560477 13540 tablet_copy_client.cc:783] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:35.560544 10491 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.560801 13540 tablet_copy_client.cc:647] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:35.561131 13495 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.562878 10620 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.563438 13540 tablet_copy_client.cc:523] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:12:35.563570 10298 consensus_peers.cc:485] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d188d415f63f448195ae90e8e3d66e0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.565721 13495 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.567955 13540 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:35.572929 10491 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.576432 10492 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:35.578675 10752 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.579653 10749 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:35.580209 13477 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:12:35.581118 10425 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f27eb42e75247c2aa142d3028e6ca2a. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:35.581401 10427 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.582551 13540 log.cc:824] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:35.582453 10220 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.584000 10360 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:35.584055 10749 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.586591 13473 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:35.582746 13477 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.595315 10360 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.596442 10490 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
W20240501 08:12:35.598217 10427 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f27eb42e75247c2aa142d3028e6ca2a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.604707 10212 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.613020 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet af24b8ab4d0c43ef986d89d4aefc263c from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47327)
I20240501 08:12:35.613276 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet af24b8ab4d0c43ef986d89d4aefc263c from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47327: session id = bb2f0099fb794400b5c24927e213e5d8-af24b8ab4d0c43ef986d89d4aefc263c
I20240501 08:12:35.616626 10772 tablet_copy_source_session.cc:205] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:35.622105 13558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af24b8ab4d0c43ef986d89d4aefc263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.631772 10619 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:35.644392 10360 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.645033 13513 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.648476 10749 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:35.652457 10620 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.653784 10492 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.654258 13405 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.654695 12992 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.656796 13558 tablet_copy_client.cc:783] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:35.656971 10362 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.657284 13558 tablet_copy_client.cc:647] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:35.658212 13612 ts_tablet_manager.cc:902] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
W20240501 08:12:35.658386 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.659114 13487 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.663143 13558 tablet_copy_client.cc:523] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:35.664458 13405 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.671444 13558 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:35.671331 10362 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:12:35.675356 10685 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af24b8ab4d0c43ef986d89d4aefc263c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.676074 13513 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.681437 10360 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.691982 10215 catalog_manager.cc:5484] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.699249 10212 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.703972 10619 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.710881 10490 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:12:35.716430 10425 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c665d7d20360440a9103e245f5f0900c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.717677 10749 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:12:35.718417 10427 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c665d7d20360440a9103e245f5f0900c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.718748 13592 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.718802 10360 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:35.722239 13619 ts_tablet_manager.cc:902] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:35.723918 10749 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.724448 12873 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:12:35.725060 10297 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae946c165cd34853917e5c8a244b6408. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.725984 10620 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.726509 13592 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.726954 10751 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.731442 12873 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.735098 10619 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:35.734634 12873 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.735706 10220 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.736316 13592 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:35.738353 10619 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.744050 10620 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.749243 10749 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:35.753152 10752 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:35.753643 10492 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:35.756470 10688 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af24b8ab4d0c43ef986d89d4aefc263c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.757077 10360 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:35.759214 10622 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:35.759531 13253 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:12:35.761648 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.762604 10212 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:35.767537 10685 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af24b8ab4d0c43ef986d89d4aefc263c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.768560 13253 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:35.771602 13610 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.772428 10622 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:35.775827 10362 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
W20240501 08:12:35.790047 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03d0f89e1d8f40fe84d275b117d8145f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.794929 10220 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 5 to 6, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
W20240501 08:12:35.808620 10297 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae946c165cd34853917e5c8a244b6408. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.826584 13612 tablet_copy_client.cc:308] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:35.847026 13558 log.cc:824] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:35.854146 13619 tablet_copy_client.cc:308] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:35.854856 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet c665d7d20360440a9103e245f5f0900c from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:37801)
I20240501 08:12:35.855048 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet c665d7d20360440a9103e245f5f0900c from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:37801: session id = c9f609481ded4d1aad9dd5f2341fd931-c665d7d20360440a9103e245f5f0900c
I20240501 08:12:35.855693 10752 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:35.858160 10512 tablet_copy_source_session.cc:205] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:35.858924 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet cf1f28d15c094fd98fac1b91d2785744 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:42583)
I20240501 08:12:35.859210 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet cf1f28d15c094fd98fac1b91d2785744 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:42583: session id = b6a964e54f3b44eda6d0666e0c1e4bea-cf1f28d15c094fd98fac1b91d2785744
I20240501 08:12:35.861243 13619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c665d7d20360440a9103e245f5f0900c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.863988 10642 tablet_copy_source_session.cc:205] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:35.865969 10492 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:35.869932 13612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1f28d15c094fd98fac1b91d2785744. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:35.870450 10425 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c665d7d20360440a9103e245f5f0900c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.870568 10622 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:35.871457 10751 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:35.872053 13477 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:12:35.872331 10427 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c665d7d20360440a9103e245f5f0900c. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:35.872524 10427 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c665d7d20360440a9103e245f5f0900c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.872583 13477 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:35.875183 13477 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.879978 10622 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.881122 13619 tablet_copy_client.cc:783] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:35.881403 13619 tablet_copy_client.cc:647] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:35.885289 13619 tablet_copy_client.cc:523] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:35.890769 13619 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:35.892877 10751 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.897437 13612 tablet_copy_client.cc:783] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:35.898736 13612 tablet_copy_client.cc:647] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:35.904438 13612 tablet_copy_client.cc:523] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:35.910550 13612 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:35.912696 10360 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:35.913089 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a6ed4ea6d6941d2af0f81eaf744bcea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.913451 13631 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.913807 10749 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:35.914685 13658 ts_tablet_manager.cc:902] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:35.918972 10220 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.923153 10619 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.923825 13631 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.932250 13624 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.938162 10362 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:35.944726 10212 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:35.945583 10360 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
W20240501 08:12:35.954607 10295 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92153666aebe47ddb080278cf24d1570. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:35.955045 10751 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.955582 13620 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.968170 10490 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:35.968910 13620 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:35.983743 13667 ts_tablet_manager.cc:902] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:35.991761 12873 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.992480 10492 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.992960 10619 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:35.994961 10752 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:35.999135 10215 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:36.000727 13658 tablet_copy_client.cc:308] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:36.001462 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet bfbef7af4781448bb04b107b53d58ca0 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:37801)
I20240501 08:12:36.001623 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet bfbef7af4781448bb04b107b53d58ca0 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:37801: session id = c9f609481ded4d1aad9dd5f2341fd931-bfbef7af4781448bb04b107b53d58ca0
I20240501 08:12:36.004617 10512 tablet_copy_source_session.cc:205] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.005887 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbef7af4781448bb04b107b53d58ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.005880 13657 ts_tablet_manager.cc:902] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
W20240501 08:12:36.009012 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50b7150bbeb645d1b00d1a561c3276d1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.009124 10492 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.009724 13624 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.011624 10360 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.012183 13624 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.017270 13659 ts_tablet_manager.cc:902] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:36.031056 13612 log.cc:824] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:36.032451 13684 ts_tablet_manager.cc:902] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:36.033854 13624 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.038422 10360 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.040016 10490 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.042215 10220 catalog_manager.cc:5484] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.056689 13540 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:36.057379 13658 tablet_copy_client.cc:783] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.057626 13658 tablet_copy_client.cc:647] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.059592 13658 tablet_copy_client.cc:523] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.060447 13540 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:36.060995 13540 ts_tablet_manager.cc:1367] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.493s	user 0.077s	sys 0.011s
I20240501 08:12:36.063875 13667 tablet_copy_client.cc:308] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:36.066623 13658 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:36.082082 13540 raft_consensus.cc:348] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:36.082670 13540 raft_consensus.cc:729] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:36.083290 13540 consensus_queue.cc:260] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:36.104544 13540 ts_tablet_manager.cc:1398] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.043s	user 0.012s	sys 0.002s
I20240501 08:12:36.106639 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-8f27eb42e75247c2aa142d3028e6ca2a received from {username='slave'} at 127.2.248.136:37801
I20240501 08:12:36.106818 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-8f27eb42e75247c2aa142d3028e6ca2a on tablet 8f27eb42e75247c2aa142d3028e6ca2a with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:36.122516 13710 ts_tablet_manager.cc:902] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:36.123515 13711 ts_tablet_manager.cc:902] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:36.130546 13684 tablet_copy_client.cc:308] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:36.131398 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet bbef0147276d410da3d3830e682efe6d from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55689)
I20240501 08:12:36.131611 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet bbef0147276d410da3d3830e682efe6d from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55689: session id = 441e281fafbf42f8938c3f2f42ff7527-bbef0147276d410da3d3830e682efe6d
I20240501 08:12:36.135845 10642 tablet_copy_source_session.cc:205] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.141335 13667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbef0147276d410da3d3830e682efe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.143256 13657 tablet_copy_client.cc:308] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:36.144258 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 6624441a1f1143648913595f905f8039 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55689)
I20240501 08:12:36.144415 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 6624441a1f1143648913595f905f8039 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55689: session id = 441e281fafbf42f8938c3f2f42ff7527-6624441a1f1143648913595f905f8039
I20240501 08:12:36.147811 10642 tablet_copy_source_session.cc:205] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.149487 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.167016 13726 ts_tablet_manager.cc:902] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:36.177677 13659 tablet_copy_client.cc:308] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:36.185170 10619 consensus_queue.cc:237] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:36.189039 10622 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:36.193507 10492 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:36.194617 13735 ts_tablet_manager.cc:902] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
W20240501 08:12:36.195399 10558 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6624441a1f1143648913595f905f8039. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.195979 10750 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:36.196627 13513 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:12:36.197350 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6624441a1f1143648913595f905f8039. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.197587 13513 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:36.201886 13711 tablet_copy_client.cc:308] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:36.202360 13513 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.204772 13710 tablet_copy_client.cc:308] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:36.203773 12667 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.215770 10492 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.217139 10360 consensus_queue.cc:237] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:36.221421 13540 ts_tablet_manager.cc:902] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:36.227566 10750 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.232088 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 8f27eb42e75247c2aa142d3028e6ca2a from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:45463)
I20240501 08:12:36.232381 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 8f27eb42e75247c2aa142d3028e6ca2a from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:45463: session id = b6a964e54f3b44eda6d0666e0c1e4bea-8f27eb42e75247c2aa142d3028e6ca2a
I20240501 08:12:36.235421 10512 tablet_copy_source_session.cc:205] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.237918 10382 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet d188d415f63f448195ae90e8e3d66e0e from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55933)
I20240501 08:12:36.238217 10382 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet d188d415f63f448195ae90e8e3d66e0e from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55933: session id = 441e281fafbf42f8938c3f2f42ff7527-d188d415f63f448195ae90e8e3d66e0e
I20240501 08:12:36.247051 13748 ts_tablet_manager.cc:902] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:36.248526 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet d188d415f63f448195ae90e8e3d66e0e from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:46779)
I20240501 08:12:36.248777 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet d188d415f63f448195ae90e8e3d66e0e from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:46779: session id = b6a964e54f3b44eda6d0666e0c1e4bea-d188d415f63f448195ae90e8e3d66e0e
I20240501 08:12:36.249368 10382 tablet_copy_source_session.cc:205] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.251833 13684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188d415f63f448195ae90e8e3d66e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.252703 10381 tablet_copy_source_session.cc:205] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.257537 13711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27eb42e75247c2aa142d3028e6ca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.258877 10492 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.259474 13513 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.260931 10220 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:36.263787 10295 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e56f5f3cd9f4a56a3d43543ac0e5d14. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.264712 10490 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.265275 12873 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.266268 10752 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.268045 10750 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.268805 12873 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.279844 13758 ts_tablet_manager.cc:902] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
W20240501 08:12:36.289999 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa6139020ac7458aa8a7ac08f63e6f26. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.291011 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188d415f63f448195ae90e8e3d66e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.294389 10619 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:36.247344 13658 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:36.296605 13658 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:36.296919 13658 ts_tablet_manager.cc:1367] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.230s	user 0.056s	sys 0.000s
I20240501 08:12:36.297923 13658 raft_consensus.cc:348] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:36.298182 13658 raft_consensus.cc:729] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:36.298460 13735 tablet_copy_client.cc:308] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:36.298394 13658 consensus_queue.cc:260] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:36.298964 13658 ts_tablet_manager.cc:1398] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:36.299837 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-bfbef7af4781448bb04b107b53d58ca0 received from {username='slave'} at 127.2.248.136:37801
I20240501 08:12:36.299978 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-bfbef7af4781448bb04b107b53d58ca0 on tablet bfbef7af4781448bb04b107b53d58ca0 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:36.299589 13740 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:12:36.303139 13513 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.304472 12873 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:36.308622 10749 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:36.313956 10221 catalog_manager.cc:5484] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.318091 10215 catalog_manager.cc:5484] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.323516 10492 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.338194 10491 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:36.340296 13758 tablet_copy_client.cc:308] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:36.346212 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet af24b8ab4d0c43ef986d89d4aefc263c from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:51519)
I20240501 08:12:36.346401 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet af24b8ab4d0c43ef986d89d4aefc263c from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:51519: session id = b6a964e54f3b44eda6d0666e0c1e4bea-af24b8ab4d0c43ef986d89d4aefc263c
I20240501 08:12:36.347436 13710 tablet_copy_client.cc:783] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.348261 13710 tablet_copy_client.cc:647] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.347661 13711 tablet_copy_client.cc:783] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.348958 13711 tablet_copy_client.cc:647] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.350129 10772 tablet_copy_source_session.cc:205] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.353612 13735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af24b8ab4d0c43ef986d89d4aefc263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.357497 13711 tablet_copy_client.cc:523] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.361191 13711 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:36.364354 13748 tablet_copy_client.cc:308] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:36.369635 13726 tablet_copy_client.cc:308] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:36.379976 13710 tablet_copy_client.cc:523] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.388803 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet b72323b8b99f4fd1aedcce69089f6204 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:43581)
I20240501 08:12:36.389008 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet b72323b8b99f4fd1aedcce69089f6204 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:43581: session id = 441e281fafbf42f8938c3f2f42ff7527-b72323b8b99f4fd1aedcce69089f6204
I20240501 08:12:36.395056 10772 tablet_copy_source_session.cc:205] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.396857 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72323b8b99f4fd1aedcce69089f6204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.400576 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 03d0f89e1d8f40fe84d275b117d8145f from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56369)
I20240501 08:12:36.400797 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 03d0f89e1d8f40fe84d275b117d8145f from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56369: session id = c9f609481ded4d1aad9dd5f2341fd931-03d0f89e1d8f40fe84d275b117d8145f
I20240501 08:12:36.410285 10642 tablet_copy_source_session.cc:205] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.412853 13748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d0f89e1d8f40fe84d275b117d8145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.425948 13748 tablet_copy_client.cc:783] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.427987 13748 tablet_copy_client.cc:647] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.430238 13659 tablet_copy_client.cc:783] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.430583 13659 tablet_copy_client.cc:647] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.430850 13748 tablet_copy_client.cc:523] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.435223 13659 tablet_copy_client.cc:523] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.443069 13748 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:36.443717 10622 consensus_queue.cc:237] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:36.464862 13558 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:36.465442 13558 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:36.465915 13558 ts_tablet_manager.cc:1367] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.795s	user 0.078s	sys 0.004s
I20240501 08:12:36.466928 13735 tablet_copy_client.cc:783] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.467280 13735 tablet_copy_client.cc:647] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.471411 13684 tablet_copy_client.cc:783] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.471755 13684 tablet_copy_client.cc:647] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.477213 10490 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:36.480913 13657 tablet_copy_client.cc:783] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.481264 13657 tablet_copy_client.cc:647] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.485440 13657 tablet_copy_client.cc:523] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.489499 13619 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:36.489774 13667 tablet_copy_client.cc:783] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.490006 13619 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:36.490042 13667 tablet_copy_client.cc:647] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.490366 13619 ts_tablet_manager.cc:1367] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.600s	user 0.047s	sys 0.011s
I20240501 08:12:36.491237 13619 raft_consensus.cc:348] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:36.491509 13619 raft_consensus.cc:729] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:36.491716 13619 consensus_queue.cc:260] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:36.492061 13667 tablet_copy_client.cc:523] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.494834 13684 tablet_copy_client.cc:523] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.496835 13558 raft_consensus.cc:348] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:36.497332 13558 raft_consensus.cc:729] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:36.497936 13558 consensus_queue.cc:260] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:36.505338 10752 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.506122 10752 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.515452 13774 ts_tablet_manager.cc:902] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:36.502578 13789 ts_tablet_manager.cc:902] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:36.503954 13786 ts_tablet_manager.cc:902] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:36.523079 13540 tablet_copy_client.cc:308] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:36.524609 13788 ts_tablet_manager.cc:902] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:36.525839 13735 tablet_copy_client.cc:523] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.532182 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet ae946c165cd34853917e5c8a244b6408 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56961)
I20240501 08:12:36.532397 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet ae946c165cd34853917e5c8a244b6408 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56961: session id = c9f609481ded4d1aad9dd5f2341fd931-ae946c165cd34853917e5c8a244b6408
I20240501 08:12:36.532517 13794 ts_tablet_manager.cc:902] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:36.535158 10381 tablet_copy_source_session.cc:205] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.535454 13619 ts_tablet_manager.cc:1398] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.045s	user 0.003s	sys 0.001s
I20240501 08:12:36.536638 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-c665d7d20360440a9103e245f5f0900c received from {username='slave'} at 127.2.248.136:37801
I20240501 08:12:36.536756 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-c665d7d20360440a9103e245f5f0900c on tablet c665d7d20360440a9103e245f5f0900c with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:36.540346 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae946c165cd34853917e5c8a244b6408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.541249 10752 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:36.541934 12992 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:12:36.542380 10555 consensus_peers.cc:485] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:36.542438 10557 consensus_peers.cc:485] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf1f28d15c094fd98fac1b91d2785744. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.543751 10360 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.544389 12992 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.547713 13804 ts_tablet_manager.cc:902] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:36.554828 13795 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:36.556696 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 50870us
I20240501 08:12:36.557498 12667 raft_consensus.cc:1203] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:36.558498 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 8f27eb42e75247c2aa142d3028e6ca2a from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:48051)
I20240501 08:12:36.558636 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 8f27eb42e75247c2aa142d3028e6ca2a from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:48051: session id = bb2f0099fb794400b5c24927e213e5d8-8f27eb42e75247c2aa142d3028e6ca2a
I20240501 08:12:36.561616 10512 tablet_copy_source_session.cc:205] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.563599 12992 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.570745 13659 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:36.570883 10220 catalog_manager.cc:5484] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.575323 10362 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.578158 13710 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:36.579435 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27eb42e75247c2aa142d3028e6ca2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.585613 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet ae946c165cd34853917e5c8a244b6408 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:60051)
I20240501 08:12:36.585776 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet ae946c165cd34853917e5c8a244b6408 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:60051: session id = bb2f0099fb794400b5c24927e213e5d8-ae946c165cd34853917e5c8a244b6408
W20240501 08:12:36.586627 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b6f09b0b5e84329944360d511578ed0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.590281 10381 tablet_copy_source_session.cc:205] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.594004 13758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae946c165cd34853917e5c8a244b6408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.594728 13540 tablet_copy_client.cc:783] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.595009 13540 tablet_copy_client.cc:647] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.596143 13726 tablet_copy_client.cc:783] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.596374 13726 tablet_copy_client.cc:647] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.602042 10489 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:36.602983 10489 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.603518 13740 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:36.604980 13740 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.605808 13758 tablet_copy_client.cc:783] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.606109 13758 tablet_copy_client.cc:647] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.608211 13540 tablet_copy_client.cc:523] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.609248 13684 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:36.614145 13758 tablet_copy_client.cc:523] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.614990 13794 tablet_copy_client.cc:308] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:36.618934 13758 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:36.626456 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet c665d7d20360440a9103e245f5f0900c from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:48051)
I20240501 08:12:36.626670 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet c665d7d20360440a9103e245f5f0900c from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:48051: session id = bb2f0099fb794400b5c24927e213e5d8-c665d7d20360440a9103e245f5f0900c
I20240501 08:12:36.627640 13740 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.628861 13735 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:36.630719 10512 tablet_copy_source_session.cc:205] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.635272 13657 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:36.636023 13540 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:36.639294 10220 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.641125 13667 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:36.641611 10489 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.650096 10749 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.653398 10360 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.656070 13796 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.656805 13796 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.659261 13807 raft_consensus.cc:1050] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:36.660396 13807 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:36.676940 13612 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:36.677587 13612 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:36.680932 13612 ts_tablet_manager.cc:1367] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.770s	user 0.073s	sys 0.004s
I20240501 08:12:36.682544 10749 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.621065 13558 ts_tablet_manager.cc:1398] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.155s	user 0.011s	sys 0.000s
I20240501 08:12:36.626217 13726 tablet_copy_client.cc:523] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.653362 13794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c665d7d20360440a9103e245f5f0900c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.662792 13804 tablet_copy_client.cc:308] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:36.688279 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 50b7150bbeb645d1b00d1a561c3276d1 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47327)
I20240501 08:12:36.688455 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 50b7150bbeb645d1b00d1a561c3276d1 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47327: session id = bb2f0099fb794400b5c24927e213e5d8-50b7150bbeb645d1b00d1a561c3276d1
I20240501 08:12:36.691236 10772 tablet_copy_source_session.cc:205] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.697057 13619 ts_tablet_manager.cc:902] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:36.703334 13774 tablet_copy_client.cc:308] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:36.706727 13794 tablet_copy_client.cc:783] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.707031 13794 tablet_copy_client.cc:647] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.707715 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet af24b8ab4d0c43ef986d89d4aefc263c from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:43581)
I20240501 08:12:36.708089 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet af24b8ab4d0c43ef986d89d4aefc263c from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:43581: session id = 441e281fafbf42f8938c3f2f42ff7527-af24b8ab4d0c43ef986d89d4aefc263c
I20240501 08:12:36.711902 13824 ts_tablet_manager.cc:902] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:36.712044 10772 tablet_copy_source_session.cc:205] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.713189 13726 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:36.715922 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b7150bbeb645d1b00d1a561c3276d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.718396 13667 log.cc:824] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:36.722648 13774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af24b8ab4d0c43ef986d89d4aefc263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.729467 10220 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.731029 10360 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.745270 13612 raft_consensus.cc:348] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:36.745829 13612 raft_consensus.cc:729] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:36.746418 13612 consensus_queue.cc:260] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:36.750010 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-af24b8ab4d0c43ef986d89d4aefc263c received from {username='slave'} at 127.2.248.133:47327
I20240501 08:12:36.750294 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-af24b8ab4d0c43ef986d89d4aefc263c on tablet af24b8ab4d0c43ef986d89d4aefc263c with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:36.751051 10490 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
W20240501 08:12:36.752826 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 185a62881ea84131a0c14987e4296338. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.754936 13794 tablet_copy_client.cc:523] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.766695 13856 ts_tablet_manager.cc:902] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:36.771051 10360 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:36.772153 10362 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:36.775419 12667 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:36.776086 13810 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:12:36.776995 10425 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.777287 13807 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:36.778333 10749 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:36.778867 13807 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:36.785773 13612 ts_tablet_manager.cc:1398] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.105s	user 0.009s	sys 0.000s
I20240501 08:12:36.786885 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-cf1f28d15c094fd98fac1b91d2785744 received from {username='slave'} at 127.2.248.134:42583
I20240501 08:12:36.787070 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-cf1f28d15c094fd98fac1b91d2785744 on tablet cf1f28d15c094fd98fac1b91d2785744 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:36.789624 13789 tablet_copy_client.cc:308] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:36.790408 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet c665d7d20360440a9103e245f5f0900c from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:45463)
I20240501 08:12:36.790491 13786 tablet_copy_client.cc:308] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:36.790567 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet c665d7d20360440a9103e245f5f0900c from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:45463: session id = b6a964e54f3b44eda6d0666e0c1e4bea-c665d7d20360440a9103e245f5f0900c
I20240501 08:12:36.791204 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 92153666aebe47ddb080278cf24d1570 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:46779)
I20240501 08:12:36.791388 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 92153666aebe47ddb080278cf24d1570 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:46779: session id = b6a964e54f3b44eda6d0666e0c1e4bea-92153666aebe47ddb080278cf24d1570
I20240501 08:12:36.793633 10512 tablet_copy_source_session.cc:205] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 08:12:36.799090 10427 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.799181 13794 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:36.801632 12272 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:36.802621 13624 consensus_queue.cc:1037] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:36.807796 10381 tablet_copy_source_session.cc:205] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.807159 13810 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.810194 12272 raft_consensus.cc:858] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:36.810554 10489 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:12:36.810485 12272 raft_consensus.cc:1565] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:36.815857 13497 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:12:36.817281 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92153666aebe47ddb080278cf24d1570. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.818629 12667 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.819792 10360 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.820379 10221 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.825115 10749 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:36.827862 13495 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.829756 13774 tablet_copy_client.cc:783] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.830035 13774 tablet_copy_client.cc:647] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.831638 12873 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:36.832976 10490 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
W20240501 08:12:36.844539 10295 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92153666aebe47ddb080278cf24d1570. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.845449 13786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92153666aebe47ddb080278cf24d1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.846863 13804 tablet_copy_client.cc:783] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.846940 10749 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:36.847913 13804 tablet_copy_client.cc:647] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.847534 10220 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.848206 13788 tablet_copy_client.cc:308] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:36.848904 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47327)
I20240501 08:12:36.849062 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 8a6ed4ea6d6941d2af0f81eaf744bcea from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47327: session id = bb2f0099fb794400b5c24927e213e5d8-8a6ed4ea6d6941d2af0f81eaf744bcea
I20240501 08:12:36.851478 10750 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.853427 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c665d7d20360440a9103e245f5f0900c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.857817 10489 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:36.859400 10772 tablet_copy_source_session.cc:205] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 08:12:36.860648 10425 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.861140 10360 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:36.861681 13796 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:36.863195 13774 tablet_copy_client.cc:523] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.864300 13796 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
W20240501 08:12:36.867439 10427 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.867923 13824 tablet_copy_client.cc:308] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:36.868059 13804 tablet_copy_client.cc:523] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.868320 13788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.868700 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet aa6139020ac7458aa8a7ac08f63e6f26 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55689)
I20240501 08:12:36.868853 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet aa6139020ac7458aa8a7ac08f63e6f26 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55689: session id = 441e281fafbf42f8938c3f2f42ff7527-aa6139020ac7458aa8a7ac08f63e6f26
I20240501 08:12:36.868885 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f27eb42e75247c2aa142d3028e6ca2a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:36.871623 10642 tablet_copy_source_session.cc:205] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:36.872007 10212 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.872283 10360 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.877887 10731 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f27eb42e75247c2aa142d3028e6ca2a with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:36.889081 10362 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:36.891676 13885 tablet_replica.cc:330] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:36.892344 13885 raft_consensus.cc:2229] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:36.892858 13885 raft_consensus.cc:2258] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:36.899191 13824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6139020ac7458aa8a7ac08f63e6f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.902588 13774 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:36.903445 13804 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:36.911382 13885 ts_tablet_manager.cc:1875] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:36.919503 13786 tablet_copy_client.cc:783] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.919929 13786 tablet_copy_client.cc:647] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.926940 13711 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:36.927455 13711 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:36.928051 13711 ts_tablet_manager.cc:1367] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.567s	user 0.050s	sys 0.010s
I20240501 08:12:36.932843 13789 tablet_copy_client.cc:783] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.933307 13786 tablet_copy_client.cc:523] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.933769 13789 tablet_copy_client.cc:647] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.908432 13824 tablet_copy_client.cc:783] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.936069 13824 tablet_copy_client.cc:647] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:36.937927 13824 tablet_copy_client.cc:523] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.944859 10490 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.945452 13497 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.950367 13824 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:36.953130 13497 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.953588 13789 tablet_copy_client.cc:523] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:36.956435 13711 raft_consensus.cc:348] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:36.957572 12667 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:36.958756 13807 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:36.958107 10489 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:36.960781 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 850d4e1020e243f4b6fa113d2946846d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.961206 13711 raft_consensus.cc:729] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:36.961565 13711 consensus_queue.cc:260] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:36.962458 13711 ts_tablet_manager.cc:1398] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.029s	user 0.000s	sys 0.002s
I20240501 08:12:36.963244 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-8f27eb42e75247c2aa142d3028e6ca2a received from {username='slave'} at 127.2.248.134:45463
I20240501 08:12:36.963368 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-8f27eb42e75247c2aa142d3028e6ca2a on tablet 8f27eb42e75247c2aa142d3028e6ca2a with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:36.964700 13786 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:36.969476 10212 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.972311 13654 raft_consensus.cc:1050] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:36.973234 13654 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:36.980688 13789 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:36.982648 10619 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.983280 13497 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.985327 12667 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:36.987105 13788 tablet_copy_client.cc:783] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:36.989661 10619 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:36.992489 12272 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:36.996847 13885 ts_tablet_manager.cc:1888] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:36.997108 13885 log.cc:1196] T 8f27eb42e75247c2aa142d3028e6ca2a P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f27eb42e75247c2aa142d3028e6ca2a
I20240501 08:12:36.997668 13558 ts_tablet_manager.cc:902] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:37.003459 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet 8f27eb42e75247c2aa142d3028e6ca2a (table rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]) successfully deleted
I20240501 08:12:37.013134 13788 tablet_copy_client.cc:647] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.016041 13788 tablet_copy_client.cc:523] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.046855 10619 consensus_queue.cc:237] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:37.051136 13900 ts_tablet_manager.cc:902] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:37.052630 13619 tablet_copy_client.cc:308] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:37.053421 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 6624441a1f1143648913595f905f8039 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56369)
I20240501 08:12:37.053604 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 6624441a1f1143648913595f905f8039 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56369: session id = c9f609481ded4d1aad9dd5f2341fd931-6624441a1f1143648913595f905f8039
I20240501 08:12:37.056998 10642 tablet_copy_source_session.cc:205] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.058105 12666 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.059104 13788 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:37.063465 10360 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:37.064103 10622 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:37.064416 12665 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:37.064900 13807 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:37.065258 13796 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:37.065537 13654 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:37.068764 13558 tablet_copy_client.cc:308] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:37.077593 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 9b6f09b0b5e84329944360d511578ed0 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47709)
I20240501 08:12:37.077843 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 9b6f09b0b5e84329944360d511578ed0 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47709: session id = bb2f0099fb794400b5c24927e213e5d8-9b6f09b0b5e84329944360d511578ed0
I20240501 08:12:37.079340 10489 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.079983 13740 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:37.080757 10555 consensus_peers.cc:485] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea502ae00eff4f72a74b8d25f5a11ede. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.081434 10749 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.081485 10642 tablet_copy_source_session.cc:205] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.082159 13740 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.083372 13558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6f09b0b5e84329944360d511578ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.086006 13631 raft_consensus.cc:1050] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:37.087445 13631 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.087827 10620 consensus_queue.cc:237] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:37.090057 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b6f09b0b5e84329944360d511578ed0. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:37.091830 10558 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b6f09b0b5e84329944360d511578ed0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.092633 13619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.092957 10362 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:37.094542 12992 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:37.097712 12992 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.100920 13856 tablet_copy_client.cc:308] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:37.100736 10362 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.101552 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:46779)
I20240501 08:12:37.101722 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:46779: session id = b6a964e54f3b44eda6d0666e0c1e4bea-6e56f5f3cd9f4a56a3d43543ac0e5d14
I20240501 08:12:37.106609 10381 tablet_copy_source_session.cc:205] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.112584 10220 catalog_manager.cc:5484] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.114336 13740 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:37.116458 10749 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:37.117614 12272 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.132299 10490 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:37.135430 13796 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.139698 12992 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:37.141023 10362 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.143348 13856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.146037 10490 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.147406 10215 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:37.149101 12665 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.153584 10221 catalog_manager.cc:5484] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.167613 13856 tablet_copy_client.cc:783] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.173935 13926 ts_tablet_manager.cc:902] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:37.179287 10622 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
W20240501 08:12:37.180603 10685 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:37.184703 10688 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.185220 12272 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.185629 10362 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.185819 13631 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.186131 13610 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.186849 10619 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.187389 13631 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.194324 13558 tablet_copy_client.cc:783] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.194707 13558 tablet_copy_client.cc:647] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.181982 10489 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:37.204413 10491 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:37.212249 13610 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.220139 10621 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.222617 10360 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.230836 12274 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.232712 13558 tablet_copy_client.cc:523] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.234144 13900 tablet_copy_client.cc:308] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:37.235000 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet cf1f28d15c094fd98fac1b91d2785744 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56369)
I20240501 08:12:37.235185 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet cf1f28d15c094fd98fac1b91d2785744 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56369: session id = c9f609481ded4d1aad9dd5f2341fd931-cf1f28d15c094fd98fac1b91d2785744
I20240501 08:12:37.237963 10214 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 6 to 7, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.238437 10642 tablet_copy_source_session.cc:205] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.239624 12667 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:37.240869 13796 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:37.243541 13796 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.246558 12667 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.247860 10749 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.251322 12274 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:37.251797 13624 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:12:37.252218 10688 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.255947 13624 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.258041 10212 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.259078 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bfbef7af4781448bb04b107b53d58ca0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.259697 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af24b8ab4d0c43ef986d89d4aefc263c with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.261526 12273 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.275552 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1f28d15c094fd98fac1b91d2785744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.275642 10214 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.285790 13856 tablet_copy_client.cc:647] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
W20240501 08:12:37.286895 10685 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.290282 13558 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:37.296945 13619 tablet_copy_client.cc:783] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.297200 13619 tablet_copy_client.cc:647] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.299767 10620 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:37.301008 13944 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:37.302649 10621 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.308004 10619 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:37.308666 13796 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:37.309067 13926 tablet_copy_client.cc:308] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:37.311534 10621 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.316380 13619 tablet_copy_client.cc:523] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.303766 12405 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:37.352972 13619 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:37.357996 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet af24b8ab4d0c43ef986d89d4aefc263c with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:37.367051 10342 tablet_service.cc:1514] Processing DeleteTablet for tablet bfbef7af4781448bb04b107b53d58ca0 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:37.369385 13856 tablet_copy_client.cc:523] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.373790 13856 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:37.412710 13796 raft_consensus.cc:1050] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:37.413653 13796 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.420745 13963 tablet_replica.cc:330] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:37.421340 13963 raft_consensus.cc:2229] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:37.422009 13963 raft_consensus.cc:2258] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:37.424253 13963 ts_tablet_manager.cc:1875] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:37.424844 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 185a62881ea84131a0c14987e4296338 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:48525)
I20240501 08:12:37.425012 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 185a62881ea84131a0c14987e4296338 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:48525: session id = 441e281fafbf42f8938c3f2f42ff7527-185a62881ea84131a0c14987e4296338
I20240501 08:12:37.427774 10512 tablet_copy_source_session.cc:205] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.429488 13926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185a62881ea84131a0c14987e4296338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.431347 10359 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.434211 10489 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:37.434370 10621 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:37.434713 13495 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:37.435917 13495 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:37.436151 13926 tablet_copy_client.cc:783] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.436419 13926 tablet_copy_client.cc:647] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.437369 12873 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
W20240501 08:12:37.438199 10298 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 850d4e1020e243f4b6fa113d2946846d. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:37.438421 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 850d4e1020e243f4b6fa113d2946846d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.440243 12403 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.440950 10750 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:37.442479 13810 consensus_queue.cc:1037] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.442920 10621 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.449452 10489 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.450428 10212 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.458962 12992 raft_consensus.cc:1050] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:37.460074 12992 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:37.462152 10752 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:37.462735 13382 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:37.463115 13926 tablet_copy_client.cc:523] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.466682 13926 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:37.467679 10489 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:37.468207 13382 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:37.470907 13382 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:37.476186 10752 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:37.478451 13726 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.478859 13726 tablet_bootstrap.cc:492] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:37.479146 13726 ts_tablet_manager.cc:1367] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.779s	user 0.048s	sys 0.014s
I20240501 08:12:37.480191 13726 raft_consensus.cc:348] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.480502 13726 raft_consensus.cc:729] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:37.480697 13726 consensus_queue.cc:260] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.482235 13726 ts_tablet_manager.cc:1398] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:12:37.483040 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-8f27eb42e75247c2aa142d3028e6ca2a received from {username='slave'} at 127.2.248.133:48051
I20240501 08:12:37.483161 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-8f27eb42e75247c2aa142d3028e6ca2a on tablet 8f27eb42e75247c2aa142d3028e6ca2a with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:37.483876 10220 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.484515 12403 raft_consensus.cc:858] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
W20240501 08:12:37.484833 12403 raft_consensus.cc:1565] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.485024 12404 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:37.491367 13382 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:37.494120 12404 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:37.495398 10489 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:37.504544 10685 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd01dd912e1e4ca9a104270f3cd004b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.506335 13794 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.506808 13794 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
W20240501 08:12:37.507038 10557 consensus_peers.cc:485] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf1f28d15c094fd98fac1b91d2785744. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.507093 13794 ts_tablet_manager.cc:1367] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.746s	user 0.048s	sys 0.012s
I20240501 08:12:37.516155 13900 tablet_copy_client.cc:783] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.516561 13900 tablet_copy_client.cc:647] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.518229 13900 tablet_copy_client.cc:523] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.518244 10621 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.518802 13944 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.521803 13900 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:37.526096 13748 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.526584 13748 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:37.526958 13748 ts_tablet_manager.cc:1367] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 1.084s	user 0.058s	sys 0.013s
I20240501 08:12:37.528007 13748 raft_consensus.cc:348] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:37.529691 13748 raft_consensus.cc:729] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:37.529963 13748 consensus_queue.cc:260] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:37.535900 13804 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.536366 13804 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:37.537569 12403 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.538363 10361 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.539070 13944 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.542140 13944 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:37.545024 10622 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:37.532104 13748 ts_tablet_manager.cc:1398] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:12:37.546965 13963 ts_tablet_manager.cc:1888] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:37.547333 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-03d0f89e1d8f40fe84d275b117d8145f received from {username='slave'} at 127.2.248.136:56369
I20240501 08:12:37.547488 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-03d0f89e1d8f40fe84d275b117d8145f on tablet 03d0f89e1d8f40fe84d275b117d8145f with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:37.548058 13963 log.cc:1196] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/af24b8ab4d0c43ef986d89d4aefc263c
I20240501 08:12:37.550271 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet af24b8ab4d0c43ef986d89d4aefc263c (table rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]) successfully deleted
I20240501 08:12:37.552047 13807 raft_consensus.cc:1050] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:37.552480 13963 tablet_replica.cc:330] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:37.552994 13807 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.553611 13963 raft_consensus.cc:2229] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:37.557694 10212 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:37.560422 13963 raft_consensus.cc:2258] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:37.560935 13804 ts_tablet_manager.cc:1367] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.658s	user 0.050s	sys 0.008s
I20240501 08:12:37.560773 12404 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.562036 13963 ts_tablet_manager.cc:1875] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:37.555881 12272 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.562788 13654 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.565197 12272 raft_consensus.cc:858] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
W20240501 08:12:37.565323 12272 raft_consensus.cc:1565] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.507943 13794 raft_consensus.cc:348] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.569845 13794 raft_consensus.cc:729] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:37.570380 13794 consensus_queue.cc:260] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
W20240501 08:12:37.571378 10427 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:37.572641 10425 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.573335 12665 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.573726 10750 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.574218 13794 ts_tablet_manager.cc:1398] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.067s	user 0.005s	sys 0.000s
I20240501 08:12:37.574193 13654 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.574697 13654 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.575641 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-c665d7d20360440a9103e245f5f0900c received from {username='slave'} at 127.2.248.133:48051
I20240501 08:12:37.575765 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-c665d7d20360440a9103e245f5f0900c on tablet c665d7d20360440a9103e245f5f0900c with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:37.578267 10621 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.578794 13654 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.582919 13711 ts_tablet_manager.cc:902] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:37.583521 13968 ts_tablet_manager.cc:902] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:37.579289 13804 raft_consensus.cc:348] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.588459 13804 raft_consensus.cc:729] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:37.593643 12405 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.595014 13796 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.600045 10359 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:37.604166 13804 consensus_queue.cc:260] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.593298 12272 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.609496 12665 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.610356 10361 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.610852 13810 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.612962 13810 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.613647 13654 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.614722 10361 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.618948 13796 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.619307 12404 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.619761 10212 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.620764 13804 ts_tablet_manager.cc:1398] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.042s	user 0.002s	sys 0.000s
I20240501 08:12:37.621491 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-50b7150bbeb645d1b00d1a561c3276d1 received from {username='slave'} at 127.2.248.133:47327
I20240501 08:12:37.621608 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-50b7150bbeb645d1b00d1a561c3276d1 on tablet 50b7150bbeb645d1b00d1a561c3276d1 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:37.627234 10489 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:37.623488 10750 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.650490 13963 ts_tablet_manager.cc:1888] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:37.653532 13735 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.654050 13735 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:37.654361 13735 ts_tablet_manager.cc:1367] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 1.060s	user 0.061s	sys 0.008s
I20240501 08:12:37.653873 12665 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.655253 13735 raft_consensus.cc:348] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.663594 13711 tablet_copy_client.cc:308] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:37.666388 12405 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.667143 13983 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.670315 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet ea502ae00eff4f72a74b8d25f5a11ede from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:42583)
I20240501 08:12:37.670511 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet ea502ae00eff4f72a74b8d25f5a11ede from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:42583: session id = b6a964e54f3b44eda6d0666e0c1e4bea-ea502ae00eff4f72a74b8d25f5a11ede
I20240501 08:12:37.674055 10642 tablet_copy_source_session.cc:205] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.673182 10212 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.675352 10622 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.676584 12272 raft_consensus.cc:1203] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:37.677253 12665 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.679528 13654 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.686041 13963 log.cc:1196] T bfbef7af4781448bb04b107b53d58ca0 P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/bfbef7af4781448bb04b107b53d58ca0
I20240501 08:12:37.686933 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet bfbef7af4781448bb04b107b53d58ca0 (table rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]) successfully deleted
I20240501 08:12:37.687485 13735 raft_consensus.cc:729] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:37.687091 13654 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.687747 13735 consensus_queue.cc:260] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.690889 13711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea502ae00eff4f72a74b8d25f5a11ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.691077 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f27eb42e75247c2aa142d3028e6ca2a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.691967 10359 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.692276 12405 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.694891 13810 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.695987 13735 ts_tablet_manager.cc:1398] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.041s	user 0.004s	sys 0.000s
I20240501 08:12:37.696249 13710 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.696583 12665 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.697758 10220 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:37.699312 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-af24b8ab4d0c43ef986d89d4aefc263c received from {username='slave'} at 127.2.248.134:51519
I20240501 08:12:37.699434 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-af24b8ab4d0c43ef986d89d4aefc263c on tablet af24b8ab4d0c43ef986d89d4aefc263c with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:37.699242 13710 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:37.704883 13710 ts_tablet_manager.cc:1367] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 1.228s	user 0.047s	sys 0.019s
I20240501 08:12:37.705562 10359 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.706179 13710 raft_consensus.cc:348] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.706957 13710 raft_consensus.cc:729] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:37.707218 13710 consensus_queue.cc:260] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.710665 12665 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.717357 13710 ts_tablet_manager.cc:1398] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.012s	user 0.001s	sys 0.001s
I20240501 08:12:37.718057 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-d188d415f63f448195ae90e8e3d66e0e received from {username='slave'} at 127.2.248.134:46779
I20240501 08:12:37.718214 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-d188d415f63f448195ae90e8e3d66e0e on tablet d188d415f63f448195ae90e8e3d66e0e with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:37.718364 10489 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.720999 12667 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:37.721733 13810 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:37.722972 10621 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:37.723619 13810 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:37.725108 10750 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:37.725631 13982 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:37.726614 13810 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.728564 10749 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.729529 10489 consensus_queue.cc:237] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:37.731694 10621 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.732193 12405 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:37.732825 13983 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:37.738277 13796 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.738302 10212 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.742568 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f27eb42e75247c2aa142d3028e6ca2a with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.743335 12404 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.748879 10220 catalog_manager.cc:5484] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 10 to 11, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:37.753369 12666 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.720988 12272 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:37.762560 13982 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.768013 12272 raft_consensus.cc:858] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
W20240501 08:12:37.768343 12272 raft_consensus.cc:1565] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.772269 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f27eb42e75247c2aa142d3028e6ca2a with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:37.773613 13963 tablet_replica.cc:330] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:37.773885 13963 raft_consensus.cc:2229] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:37.774530 13963 raft_consensus.cc:2258] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:37.776275 12405 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:37.776342 13963 ts_tablet_manager.cc:1875] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:37.777024 13953 consensus_queue.cc:1037] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:37.779780 12405 raft_consensus.cc:858] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:37.779889 12405 raft_consensus.cc:1565] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.784044 12405 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.788682 13953 raft_consensus.cc:1050] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:37.789582 13953 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.790544 13789 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.790973 13789 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:37.791299 13789 ts_tablet_manager.cc:1367] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.824s	user 0.059s	sys 0.008s
I20240501 08:12:37.792075 13789 raft_consensus.cc:348] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.792392 13789 raft_consensus.cc:729] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:37.792611 13789 consensus_queue.cc:260] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:37.793067 12273 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.793725 12667 raft_consensus.cc:1261] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:37.794243 13789 ts_tablet_manager.cc:1398] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:12:37.794342 13953 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.794752 12405 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.794865 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-c665d7d20360440a9103e245f5f0900c received from {username='slave'} at 127.2.248.134:45463
I20240501 08:12:37.794984 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-c665d7d20360440a9103e245f5f0900c on tablet c665d7d20360440a9103e245f5f0900c with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:37.796185 14001 consensus_queue.cc:1035] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:12:37.797153 10688 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.797910 13949 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:37.797828 12667 raft_consensus.cc:2932] T 8f27eb42e75247c2aa142d3028e6ca2a P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.799666 10621 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.800210 13944 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.801785 12405 raft_consensus.cc:858] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:37.801891 12405 raft_consensus.cc:1565] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.802621 13949 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.803534 12405 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.808025 12273 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.814522 10619 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.814812 12405 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:37.823879 10215 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 8 to 9, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.712604 13758 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.825244 13758 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:37.825605 13758 ts_tablet_manager.cc:1367] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 1.207s	user 0.042s	sys 0.018s
I20240501 08:12:37.826445 13758 raft_consensus.cc:348] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:37.826800 13758 raft_consensus.cc:729] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:37.827165 13758 consensus_queue.cc:260] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:37.827571 12665 raft_consensus.cc:1203] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:37.835487 10750 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:37.838948 12405 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.839620 13953 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.848631 10621 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.853611 13944 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.858067 13774 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.862685 13774 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:37.870060 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c665d7d20360440a9103e245f5f0900c with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.870910 10489 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:37.871685 13711 tablet_copy_client.cc:783] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.872232 13711 tablet_copy_client.cc:647] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.873018 13774 ts_tablet_manager.cc:1367] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 0.963s	user 0.056s	sys 0.007s
I20240501 08:12:37.873875 10619 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.874325 12665 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.874526 13711 tablet_copy_client.cc:523] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.874956 14001 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.839264 12273 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.860504 12254 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f27eb42e75247c2aa142d3028e6ca2a with delete_type TABLET_DATA_TOMBSTONED (TS bb2f0099fb794400b5c24927e213e5d8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53270
I20240501 08:12:37.875769 14001 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.878505 13953 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.879675 13711 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:37.881439 13624 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.887295 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af24b8ab4d0c43ef986d89d4aefc263c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.887915 10750 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:37.888438 14001 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:37.888319 12273 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.889259 10214 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:37.890944 14001 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.896580 10749 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.861132 12272 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:37.888309 12405 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.890713 13786 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:12:37.904237 13786 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:37.904987 13786 ts_tablet_manager.cc:1367] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.951s	user 0.057s	sys 0.014s
I20240501 08:12:37.905707 10622 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:37.907609 10215 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.908632 13786 raft_consensus.cc:348] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:37.909157 13786 raft_consensus.cc:729] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:37.909699 10619 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.909351 13786 consensus_queue.cc:260] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:37.913440 12253 tablet_service.cc:1514] Processing DeleteTablet for tablet c665d7d20360440a9103e245f5f0900c with delete_type TABLET_DATA_TOMBSTONED (TS bb2f0099fb794400b5c24927e213e5d8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53270
I20240501 08:12:37.915426 13786 ts_tablet_manager.cc:1398] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.002s
I20240501 08:12:37.916600 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c665d7d20360440a9103e245f5f0900c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.917582 12665 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.919145 13856 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.919636 13856 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:37.941823 13758 ts_tablet_manager.cc:1398] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.116s	user 0.002s	sys 0.000s
I20240501 08:12:37.948522 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-92153666aebe47ddb080278cf24d1570 received from {username='slave'} at 127.2.248.134:46779
I20240501 08:12:37.949965 10489 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:37.950954 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-92153666aebe47ddb080278cf24d1570 on tablet 92153666aebe47ddb080278cf24d1570 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:37.951601 10382 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-ae946c165cd34853917e5c8a244b6408 received from {username='slave'} at 127.2.248.133:60051
I20240501 08:12:37.951757 10382 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-ae946c165cd34853917e5c8a244b6408 on tablet ae946c165cd34853917e5c8a244b6408 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:37.952247 12665 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.956540 14001 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.958673 10750 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:37.959010 13654 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.959355 14001 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:37.959650 12667 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.961084 10749 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:37.964339 13748 ts_tablet_manager.cc:902] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:37.964893 10214 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 9 to 10, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:37.965621 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c665d7d20360440a9103e245f5f0900c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:37.964392 13856 ts_tablet_manager.cc:1367] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.591s	user 0.051s	sys 0.003s
I20240501 08:12:37.972900 13856 raft_consensus.cc:348] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:37.978593 13540 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.979422 13963 ts_tablet_manager.cc:1888] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:12:37.979795 13856 raft_consensus.cc:729] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:37.983062 13657 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.983525 13657 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:37.984300 13856 consensus_queue.cc:260] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:37.985060 13963 log.cc:1196] T 8f27eb42e75247c2aa142d3028e6ca2a P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f27eb42e75247c2aa142d3028e6ca2a
I20240501 08:12:37.988499 13856 ts_tablet_manager.cc:1398] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.000s
I20240501 08:12:37.989539 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet 8f27eb42e75247c2aa142d3028e6ca2a (table rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]) successfully deleted
I20240501 08:12:37.990324 13657 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 1.360s	user 0.048s	sys 0.010s
I20240501 08:12:37.990321 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet c665d7d20360440a9103e245f5f0900c with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:37.991709 14034 tablet_replica.cc:330] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: stopping tablet replica
I20240501 08:12:37.992311 14034 raft_consensus.cc:2229] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:37.993633 14034 raft_consensus.cc:2258] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:37.995317 14034 ts_tablet_manager.cc:1875] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:37.997498 14043 tablet_replica.cc:330] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:37.997989 14043 raft_consensus.cc:2229] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:37.997825 12385 tablet_service.cc:1514] Processing DeleteTablet for tablet c665d7d20360440a9103e245f5f0900c with delete_type TABLET_DATA_TOMBSTONED (TS b6a964e54f3b44eda6d0666e0c1e4bea not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38792
I20240501 08:12:38.005983 14043 raft_consensus.cc:2258] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.015671 14046 tablet_replica.cc:330] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: stopping tablet replica
I20240501 08:12:38.016346 10382 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-6e56f5f3cd9f4a56a3d43543ac0e5d14 received from {username='slave'} at 127.2.248.134:46779
I20240501 08:12:38.016496 10382 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-6e56f5f3cd9f4a56a3d43543ac0e5d14 on tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:38.017048 14046 raft_consensus.cc:2229] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:38.021925 14043 ts_tablet_manager.cc:1875] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.024585 14046 raft_consensus.cc:2258] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:38.026137 14046 ts_tablet_manager.cc:1875] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.030828 13540 tablet_bootstrap.cc:492] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:38.038272 14034 ts_tablet_manager.cc:1888] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:38.038483 14034 log.cc:1196] T 8f27eb42e75247c2aa142d3028e6ca2a P bb2f0099fb794400b5c24927e213e5d8: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/wals/8f27eb42e75247c2aa142d3028e6ca2a
I20240501 08:12:38.039489 10207 catalog_manager.cc:4830] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): tablet 8f27eb42e75247c2aa142d3028e6ca2a (table rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]) successfully deleted
I20240501 08:12:38.039541 14034 tablet_replica.cc:330] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: stopping tablet replica
I20240501 08:12:38.039748 14034 raft_consensus.cc:2229] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:38.040057 14034 raft_consensus.cc:2258] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:38.040365 13667 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.052966 13659 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.053416 13659 tablet_bootstrap.cc:492] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:38.053819 13659 ts_tablet_manager.cc:1367] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 1.483s	user 0.061s	sys 0.000s
I20240501 08:12:38.056216 13540 ts_tablet_manager.cc:1367] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 1.395s	user 0.053s	sys 0.013s
I20240501 08:12:38.057173 13540 raft_consensus.cc:348] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:38.057575 13540 raft_consensus.cc:729] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:38.057842 13540 consensus_queue.cc:260] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:38.058950 13540 ts_tablet_manager.cc:1398] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:12:38.060554 10382 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-ae946c165cd34853917e5c8a244b6408 received from {username='slave'} at 127.2.248.136:56961
I20240501 08:12:38.060703 10382 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-ae946c165cd34853917e5c8a244b6408 on tablet ae946c165cd34853917e5c8a244b6408 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:38.064817 14034 ts_tablet_manager.cc:1875] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.066864 12516 tablet_service.cc:1514] Processing DeleteTablet for tablet af24b8ab4d0c43ef986d89d4aefc263c with delete_type TABLET_DATA_TOMBSTONED (TS 441e281fafbf42f8938c3f2f42ff7527 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:32994
I20240501 08:12:38.068187 13684 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.068634 13684 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:38.069336 13684 ts_tablet_manager.cc:1367] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 1.571s	user 0.053s	sys 0.010s
I20240501 08:12:38.073830 14017 ts_tablet_manager.cc:902] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:38.074716 13667 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:38.075062 13667 ts_tablet_manager.cc:1367] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 1.483s	user 0.059s	sys 0.005s
I20240501 08:12:38.080155 10619 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:38.083025 12405 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:38.083721 13513 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:38.087594 10749 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:38.088169 13513 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:38.089991 10489 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:38.090814 14046 ts_tablet_manager.cc:1888] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:12:38.090946 13382 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:38.091673 13382 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:38.094532 14043 ts_tablet_manager.cc:1888] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:12:38.094732 14043 log.cc:1196] T c665d7d20360440a9103e245f5f0900c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/c665d7d20360440a9103e245f5f0900c
I20240501 08:12:38.096436 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet c665d7d20360440a9103e245f5f0900c (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:38.096792 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf1f28d15c094fd98fac1b91d2785744 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.055936 13788 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.100669 13788 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:38.101028 13788 ts_tablet_manager.cc:1367] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 1.042s	user 0.040s	sys 0.016s
I20240501 08:12:38.101351 10489 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:38.101950 13788 raft_consensus.cc:348] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:38.102273 13788 raft_consensus.cc:729] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:38.102526 13788 consensus_queue.cc:260] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:38.103911 13788 ts_tablet_manager.cc:1398] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:12:38.104650 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-8a6ed4ea6d6941d2af0f81eaf744bcea received from {username='slave'} at 127.2.248.133:47327
I20240501 08:12:38.104774 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-8a6ed4ea6d6941d2af0f81eaf744bcea on tablet 8a6ed4ea6d6941d2af0f81eaf744bcea with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:38.106053 10749 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:38.108846 10220 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.097151 13856 ts_tablet_manager.cc:902] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:38.097980 12405 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:38.118741 14034 ts_tablet_manager.cc:1888] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:38.121717 13856 tablet_copy_client.cc:308] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:38.121753 13774 raft_consensus.cc:348] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.122675 13774 raft_consensus.cc:729] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:38.122610 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet cd01dd912e1e4ca9a104270f3cd004b1 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:51519)
I20240501 08:12:38.122939 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet cd01dd912e1e4ca9a104270f3cd004b1 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:51519: session id = b6a964e54f3b44eda6d0666e0c1e4bea-cd01dd912e1e4ca9a104270f3cd004b1
I20240501 08:12:38.122855 13659 raft_consensus.cc:348] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.123271 13968 tablet_copy_client.cc:308] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:38.125783 10772 tablet_copy_source_session.cc:205] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.126940 13748 tablet_copy_client.cc:308] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:38.127594 10382 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 850d4e1020e243f4b6fa113d2946846d from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56961)
I20240501 08:12:38.127763 10382 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 850d4e1020e243f4b6fa113d2946846d from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56961: session id = c9f609481ded4d1aad9dd5f2341fd931-850d4e1020e243f4b6fa113d2946846d
I20240501 08:12:38.128708 13774 consensus_queue.cc:260] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.130826 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 92153666aebe47ddb080278cf24d1570 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55933)
W20240501 08:12:38.130856 10208 catalog_manager.cc:4810] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): delete failed for tablet af24b8ab4d0c43ef986d89d4aefc263c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af24b8ab4d0c43ef986d89d4aefc263c already in progress: copying tablet
I20240501 08:12:38.130985 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 92153666aebe47ddb080278cf24d1570 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55933: session id = 441e281fafbf42f8938c3f2f42ff7527-92153666aebe47ddb080278cf24d1570
I20240501 08:12:38.129477 13657 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.135995 10619 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:38.136848 10382 tablet_copy_source_session.cc:205] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.135059 10381 tablet_copy_source_session.cc:205] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.140709 14034 log.cc:1196] T c665d7d20360440a9103e245f5f0900c P bb2f0099fb794400b5c24927e213e5d8: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/wals/c665d7d20360440a9103e245f5f0900c
I20240501 08:12:38.141311 10750 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:38.141356 12405 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:38.142014 13513 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:38.142608 13513 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:38.143204 10207 catalog_manager.cc:4830] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): tablet c665d7d20360440a9103e245f5f0900c (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:38.146713 13513 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:38.150118 13856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd01dd912e1e4ca9a104270f3cd004b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.153620 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf1f28d15c094fd98fac1b91d2785744 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.154878 12405 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:38.158663 10750 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:38.161397 13748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d4e1020e243f4b6fa113d2946846d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.161059 10220 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.169675 13667 raft_consensus.cc:348] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.169955 13667 raft_consensus.cc:729] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:38.170202 13667 consensus_queue.cc:260] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.170714 13667 ts_tablet_manager.cc:1398] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.096s	user 0.001s	sys 0.000s
I20240501 08:12:38.171526 14046 log.cc:1196] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/wals/c665d7d20360440a9103e245f5f0900c
I20240501 08:12:38.172166 13657 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:38.172497 13657 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.172708 10205 catalog_manager.cc:4830] TS b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): tablet c665d7d20360440a9103e245f5f0900c (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:38.173038 13968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92153666aebe47ddb080278cf24d1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.179387 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-bbef0147276d410da3d3830e682efe6d received from {username='slave'} at 127.2.248.135:55689
I20240501 08:12:38.179534 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-bbef0147276d410da3d3830e682efe6d on tablet bbef0147276d410da3d3830e682efe6d with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:38.181815 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1f28d15c094fd98fac1b91d2785744 with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:38.185012 13968 tablet_copy_client.cc:783] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.190372 13667 ts_tablet_manager.cc:902] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:38.191205 13558 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.191813 13558 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:38.192188 13558 ts_tablet_manager.cc:1367] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.902s	user 0.051s	sys 0.005s
I20240501 08:12:38.193195 13558 raft_consensus.cc:348] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:38.193540 13558 raft_consensus.cc:729] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:38.193771 13558 consensus_queue.cc:260] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:38.200083 13774 ts_tablet_manager.cc:1398] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.291s	user 0.002s	sys 0.007s
I20240501 08:12:38.200043 12516 tablet_service.cc:1514] Processing DeleteTablet for tablet af24b8ab4d0c43ef986d89d4aefc263c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 10)) from {username='slave'} at 127.0.0.1:32994
I20240501 08:12:38.201802 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-af24b8ab4d0c43ef986d89d4aefc263c received from {username='slave'} at 127.2.248.135:43581
I20240501 08:12:38.201958 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-af24b8ab4d0c43ef986d89d4aefc263c on tablet af24b8ab4d0c43ef986d89d4aefc263c with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:38.205157 13659 raft_consensus.cc:729] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:38.205435 13659 consensus_queue.cc:260] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.206386 13558 ts_tablet_manager.cc:1398] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.000s
I20240501 08:12:38.209444 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-9b6f09b0b5e84329944360d511578ed0 received from {username='slave'} at 127.2.248.133:47709
I20240501 08:12:38.209594 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-9b6f09b0b5e84329944360d511578ed0 on tablet 9b6f09b0b5e84329944360d511578ed0 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:38.209894 14059 tablet_replica.cc:330] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: stopping tablet replica
I20240501 08:12:38.210426 14059 raft_consensus.cc:2229] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:38.210817 14059 raft_consensus.cc:2258] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:38.210951 12404 raft_consensus.cc:1203] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:38.212324 14059 ts_tablet_manager.cc:1875] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.213941 12404 raft_consensus.cc:1203] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:38.214213 12404 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:38.217852 13748 tablet_copy_client.cc:783] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.218147 13748 tablet_copy_client.cc:647] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.218652 13968 tablet_copy_client.cc:647] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.220742 13748 tablet_copy_client.cc:523] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:38.221045 13968 tablet_copy_client.cc:523] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:38.223591 13684 raft_consensus.cc:348] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.223948 13684 raft_consensus.cc:729] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:38.224223 13684 consensus_queue.cc:260] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.224488 13748 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:38.231683 12647 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1f28d15c094fd98fac1b91d2785744 with delete_type TABLET_DATA_TOMBSTONED (TS c9f609481ded4d1aad9dd5f2341fd931 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60706
I20240501 08:12:38.231999 12405 raft_consensus.cc:1203] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240501 08:12:38.242846 10207 catalog_manager.cc:4810] TS c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): delete failed for tablet cf1f28d15c094fd98fac1b91d2785744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf1f28d15c094fd98fac1b91d2785744 already in progress: copying tablet
I20240501 08:12:38.244787 14071 tablet_replica.cc:330] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:38.245369 14071 raft_consensus.cc:2229] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:38.245846 14071 raft_consensus.cc:2258] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.255681 14071 ts_tablet_manager.cc:1875] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.260457 13856 tablet_copy_client.cc:783] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.260704 13856 tablet_copy_client.cc:647] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.266620 13856 tablet_copy_client.cc:523] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:38.276918 13856 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:38.288394 13684 ts_tablet_manager.cc:1398] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.078s	user 0.002s	sys 0.000s
I20240501 08:12:38.245635 13659 ts_tablet_manager.cc:1398] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.192s	user 0.002s	sys 0.000s
I20240501 08:12:38.290263 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-d188d415f63f448195ae90e8e3d66e0e received from {username='slave'} at 127.2.248.135:55933
I20240501 08:12:38.290438 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-d188d415f63f448195ae90e8e3d66e0e on tablet d188d415f63f448195ae90e8e3d66e0e with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:38.290786 12534 raft_consensus.cc:1203] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:38.291308 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-b72323b8b99f4fd1aedcce69089f6204 received from {username='slave'} at 127.2.248.135:43581
I20240501 08:12:38.291471 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-b72323b8b99f4fd1aedcce69089f6204 on tablet b72323b8b99f4fd1aedcce69089f6204 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:38.292547 14017 tablet_copy_client.cc:308] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:38.293649 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 9b6f09b0b5e84329944360d511578ed0 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55689)
I20240501 08:12:38.293818 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 9b6f09b0b5e84329944360d511578ed0 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55689: session id = 441e281fafbf42f8938c3f2f42ff7527-9b6f09b0b5e84329944360d511578ed0
I20240501 08:12:38.296903 10642 tablet_copy_source_session.cc:205] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.300501 13944 raft_consensus.cc:1050] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:38.301492 13944 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:38.329448 12272 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.330265 10358 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.331053 13624 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.253826 13968 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:38.331627 13624 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.335412 10489 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.336030 13953 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.337477 13382 raft_consensus.cc:1050] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:38.338523 13382 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:38.257512 14059 ts_tablet_manager.cc:1888] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:12:38.283337 13657 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.293s	user 0.009s	sys 0.003s
I20240501 08:12:38.343518 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-6624441a1f1143648913595f905f8039 received from {username='slave'} at 127.2.248.135:55689
I20240501 08:12:38.343648 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-6624441a1f1143648913595f905f8039 on tablet 6624441a1f1143648913595f905f8039 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:38.346176 12272 raft_consensus.cc:1203] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:38.352442 12667 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.353113 13740 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.357286 10361 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.357342 13619 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.357890 13619 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:38.358199 13740 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.372689 14071 ts_tablet_manager.cc:1888] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:38.375639 14071 log.cc:1196] T cf1f28d15c094fd98fac1b91d2785744 P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/cf1f28d15c094fd98fac1b91d2785744
I20240501 08:12:38.377312 13619 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 1.024s	user 0.046s	sys 0.008s
I20240501 08:12:38.378019 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet cf1f28d15c094fd98fac1b91d2785744 (table rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]) successfully deleted
I20240501 08:12:38.381775 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6f09b0b5e84329944360d511578ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.381596 13619 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:38.381989 13619 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:38.382623 13619 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:38.391204 13619 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.001s
I20240501 08:12:38.395318 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-6624441a1f1143648913595f905f8039 received from {username='slave'} at 127.2.248.136:56369
I20240501 08:12:38.395465 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-6624441a1f1143648913595f905f8039 on tablet 6624441a1f1143648913595f905f8039 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:38.410558 14059 log.cc:1196] T af24b8ab4d0c43ef986d89d4aefc263c P 441e281fafbf42f8938c3f2f42ff7527: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/wals/af24b8ab4d0c43ef986d89d4aefc263c
I20240501 08:12:38.413844 12534 raft_consensus.cc:1203] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:38.418324 13900 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.419205 13900 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:38.422309 13900 ts_tablet_manager.cc:1367] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.901s	user 0.046s	sys 0.011s
I20240501 08:12:38.423625 13900 raft_consensus.cc:348] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:38.426249 12536 raft_consensus.cc:1203] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:38.432204 13900 raft_consensus.cc:729] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:38.432618 13900 consensus_queue.cc:260] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:38.436441 12647 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1f28d15c094fd98fac1b91d2785744 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:60706
W20240501 08:12:38.437295 10207 catalog_manager.cc:4810] TS c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): delete failed for tablet cf1f28d15c094fd98fac1b91d2785744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf1f28d15c094fd98fac1b91d2785744 already in progress: copying tablet
I20240501 08:12:38.437992 13900 ts_tablet_manager.cc:1398] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.015s	user 0.001s	sys 0.000s
I20240501 08:12:38.441133 12647 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1f28d15c094fd98fac1b91d2785744 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:60706
I20240501 08:12:38.441506 14076 tablet_replica.cc:330] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: stopping tablet replica
I20240501 08:12:38.442013 14076 raft_consensus.cc:2229] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:38.442375 14076 raft_consensus.cc:2258] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:38.443338 14076 ts_tablet_manager.cc:1875] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.447260 10208 catalog_manager.cc:4830] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): tablet af24b8ab4d0c43ef986d89d4aefc263c (table rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]) successfully deleted
I20240501 08:12:38.451376 14076 ts_tablet_manager.cc:1888] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:38.451586 14076 log.cc:1196] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal/wals/cf1f28d15c094fd98fac1b91d2785744
I20240501 08:12:38.452508 10207 catalog_manager.cc:4830] TS c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): tablet cf1f28d15c094fd98fac1b91d2785744 (table rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]) successfully deleted
I20240501 08:12:38.453812 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-cf1f28d15c094fd98fac1b91d2785744 received from {username='slave'} at 127.2.248.136:56369
I20240501 08:12:38.453929 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-cf1f28d15c094fd98fac1b91d2785744 on tablet cf1f28d15c094fd98fac1b91d2785744 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:38.465543 14017 tablet_copy_client.cc:783] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.465814 14017 tablet_copy_client.cc:647] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.471385 13667 tablet_copy_client.cc:308] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:38.472359 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 850d4e1020e243f4b6fa113d2946846d from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55933)
I20240501 08:12:38.472568 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 850d4e1020e243f4b6fa113d2946846d from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55933: session id = 441e281fafbf42f8938c3f2f42ff7527-850d4e1020e243f4b6fa113d2946846d
I20240501 08:12:38.475220 14017 tablet_copy_client.cc:523] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:38.475334 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:38.476397 10381 tablet_copy_source_session.cc:205] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.480815 13667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d4e1020e243f4b6fa113d2946846d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.491644 14017 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:38.498765 13711 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.499287 13711 tablet_bootstrap.cc:492] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:38.499598 13711 ts_tablet_manager.cc:1367] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.620s	user 0.058s	sys 0.004s
I20240501 08:12:38.500631 13711 raft_consensus.cc:348] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:38.500926 13711 raft_consensus.cc:729] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:38.501124 13711 consensus_queue.cc:260] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:38.501652 13711 ts_tablet_manager.cc:1398] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:38.502465 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-ea502ae00eff4f72a74b8d25f5a11ede received from {username='slave'} at 127.2.248.134:42583
I20240501 08:12:38.502593 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-ea502ae00eff4f72a74b8d25f5a11ede on tablet ea502ae00eff4f72a74b8d25f5a11ede with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:38.507838 10491 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:38.507668 13624 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:38.515748 12272 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
W20240501 08:12:38.517755 14094 log.cc:924] Time spent T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Append to log took a long time: real 0.183s	user 0.000s	sys 0.000s
I20240501 08:12:38.518612 10220 catalog_manager.cc:5484] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.523804 13748 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.525995 13748 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:38.526301 13748 ts_tablet_manager.cc:1367] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.302s	user 0.050s	sys 0.004s
I20240501 08:12:38.529009 12665 raft_consensus.cc:1203] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:38.527807 13748 raft_consensus.cc:348] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.530242 13748 raft_consensus.cc:729] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:38.532910 13748 consensus_queue.cc:260] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.535413 10358 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:38.535949 13382 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:38.541508 13748 ts_tablet_manager.cc:1398] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.015s	user 0.001s	sys 0.001s
I20240501 08:12:38.541718 12665 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:38.543449 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-850d4e1020e243f4b6fa113d2946846d received from {username='slave'} at 127.2.248.136:56961
I20240501 08:12:38.543604 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-850d4e1020e243f4b6fa113d2946846d on tablet 850d4e1020e243f4b6fa113d2946846d with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:38.548045 10750 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:38.550756 10358 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:38.553710 10217 catalog_manager.cc:5484] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:38.559525 12272 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.560380 13953 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.562880 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:38.563302 13953 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:38.566711 10490 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.566924 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50b7150bbeb645d1b00d1a561c3276d1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.568025 12273 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:38.568696 10217 catalog_manager.cc:5484] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.575536 10489 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.576798 12273 raft_consensus.cc:1203] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:38.577368 13944 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.579414 10489 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:38.581506 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b7150bbeb645d1b00d1a561c3276d1 with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:38.588311 12667 raft_consensus.cc:1203] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:38.593864 13740 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.603701 14125 tablet_replica.cc:330] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:38.604090 14125 raft_consensus.cc:2229] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:38.604590 14125 raft_consensus.cc:2258] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.617453 14125 ts_tablet_manager.cc:1875] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.506037 13667 tablet_copy_client.cc:783] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.622398 13824 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.651255 10489 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:38.652186 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:38.656483 13667 tablet_copy_client.cc:647] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.661115 12667 raft_consensus.cc:1203] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:38.663327 13824 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:38.663772 13824 ts_tablet_manager.cc:1367] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 1.713s	user 0.055s	sys 0.002s
I20240501 08:12:38.667502 13824 raft_consensus.cc:348] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.670055 13824 raft_consensus.cc:729] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:38.670342 13824 consensus_queue.cc:260] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.670830 12273 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:38.681856 13824 ts_tablet_manager.cc:1398] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.002s
I20240501 08:12:38.683790 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-aa6139020ac7458aa8a7ac08f63e6f26 received from {username='slave'} at 127.2.248.135:55689
I20240501 08:12:38.683933 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-aa6139020ac7458aa8a7ac08f63e6f26 on tablet aa6139020ac7458aa8a7ac08f63e6f26 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:38.691277 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:38.701728 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:38.703801 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:38.714908 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:38.718809 13667 tablet_copy_client.cc:523] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:38.712004 14125 ts_tablet_manager.cc:1888] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:38.726291 14125 log.cc:1196] T 50b7150bbeb645d1b00d1a561c3276d1 P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/50b7150bbeb645d1b00d1a561c3276d1
I20240501 08:12:38.727109 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet 50b7150bbeb645d1b00d1a561c3276d1 (table rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]) successfully deleted
I20240501 08:12:38.744199 13969 raft_consensus.cc:1050] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:38.745293 13969 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.749135 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:38.767372 10489 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.767928 13986 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.768168 12534 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.769088 13497 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.771927 13667 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:38.775904 14124 raft_consensus.cc:1050] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:38.776726 14124 consensus_queue.cc:237] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:38.777961 10750 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.784265 13986 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.805590 12535 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.805111 12536 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:38.806340 13944 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.809219 10619 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.810539 13944 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.814189 10359 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.814716 13944 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.820932 13986 raft_consensus.cc:1013] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: attempt to promote peer 441e281fafbf42f8938c3f2f42ff7527: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:12:38.832288 12534 raft_consensus.cc:1203] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:38.837607 13513 raft_consensus.cc:1050] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:38.838421 13513 consensus_queue.cc:237] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:38.839991 13944 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:38.843420 12404 raft_consensus.cc:1203] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:38.846338 13969 raft_consensus.cc:1050] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:38.846112 10359 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:38.847092 13969 consensus_queue.cc:237] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:38.848937 12404 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.849701 12873 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.849817 13986 raft_consensus.cc:1050] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:38.850968 13986 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:38.853137 10220 catalog_manager.cc:5484] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:38.854480 10621 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:38.884701 12536 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:38.885605 10750 consensus_queue.cc:237] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:38.888090 10359 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.888666 14124 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.891505 13953 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:38.895828 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b72323b8b99f4fd1aedcce69089f6204 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.899173 10220 catalog_manager.cc:5484] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:38.903410 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet b72323b8b99f4fd1aedcce69089f6204 with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:38.905526 13497 raft_consensus.cc:1050] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:38.906484 13497 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:38.927270 10359 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.927809 10489 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.928311 13740 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.928834 13740 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.953616 12534 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.954558 12535 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.955569 14166 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 08:12:38.958122 12535 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:38.965188 13513 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.970218 14168 tablet_replica.cc:330] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:38.970533 14168 raft_consensus.cc:2229] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:38.970957 14168 raft_consensus.cc:2258] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.971792 14171 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:38.977435 12667 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.977502 12273 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.978824 12404 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.987179 14168 ts_tablet_manager.cc:1875] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.989413 10619 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:38.991928 12667 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:12:38.992444 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:38.992683 13382 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.993548 10619 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.995709 10750 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.995720 10749 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.995868 13986 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.996619 10491 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.996395 10361 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:38.997598 13382 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:38.999256 13740 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.001087 10490 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.003679 12667 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.005086 14178 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 08:12:39.005941 14187 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.006426 13969 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.007722 10491 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.010288 10214 catalog_manager.cc:5484] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.014122 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d0f89e1d8f40fe84d275b117d8145f with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:39.014464 14125 tablet_replica.cc:330] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:39.014657 14125 raft_consensus.cc:2229] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.014848 14125 raft_consensus.cc:2258] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.015861 14125 ts_tablet_manager.cc:1875] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.021788 10214 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.025410 14125 ts_tablet_manager.cc:1888] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.025640 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03d0f89e1d8f40fe84d275b117d8145f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.026304 14125 log.cc:1196] T 03d0f89e1d8f40fe84d275b117d8145f P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/03d0f89e1d8f40fe84d275b117d8145f
I20240501 08:12:39.027144 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet 03d0f89e1d8f40fe84d275b117d8145f (table rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]) successfully deleted
I20240501 08:12:39.029148 10489 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.030618 14186 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.039901 13953 raft_consensus.cc:1050] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:39.040779 13953 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:39.048008 10491 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.063344 12535 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.064875 10749 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.065963 10214 catalog_manager.cc:5484] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:39.074256 10619 consensus_queue.cc:237] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:39.076795 14174 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.077313 14174 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.078099 14174 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.100092 13926 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:39.100625 13926 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:39.102948 12534 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.103897 14187 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:12:39.104202 10295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.131:34319, user_credentials={real_user=slave}} blocked reactor thread for 106962us
W20240501 08:12:39.105141 10557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.136:42153, user_credentials={real_user=slave}} blocked reactor thread for 95644.6us
I20240501 08:12:39.105299 12404 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.105894 12405 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.107030 10490 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.111423 13926 ts_tablet_manager.cc:1367] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 1.634s	user 0.054s	sys 0.016s
I20240501 08:12:39.111865 12273 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.112452 14166 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.112553 13926 raft_consensus.cc:348] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:39.112807 13667 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:39.112882 14174 raft_consensus.cc:1013] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: attempt to promote peer c9f609481ded4d1aad9dd5f2341fd931: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:12:39.113726 13926 raft_consensus.cc:729] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:39.114027 13926 consensus_queue.cc:260] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:39.116465 13926 ts_tablet_manager.cc:1398] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:12:39.117098 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-185a62881ea84131a0c14987e4296338 received from {username='slave'} at 127.2.248.135:48525
I20240501 08:12:39.117218 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-185a62881ea84131a0c14987e4296338 on tablet 185a62881ea84131a0c14987e4296338 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:39.117631 13667 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:39.118022 13667 ts_tablet_manager.cc:1367] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 0.346s	user 0.052s	sys 0.003s
I20240501 08:12:39.120180 13667 raft_consensus.cc:348] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:39.120436 13667 raft_consensus.cc:729] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:39.120363 14189 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.120637 13667 consensus_queue.cc:260] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:39.125231 12534 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.126861 13667 ts_tablet_manager.cc:1398] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.000s
I20240501 08:12:39.129559 10619 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.130290 13953 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.132961 14173 raft_consensus.cc:1050] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:39.133234 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-850d4e1020e243f4b6fa113d2946846d received from {username='slave'} at 127.2.248.135:55933
I20240501 08:12:39.133534 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-850d4e1020e243f4b6fa113d2946846d on tablet 850d4e1020e243f4b6fa113d2946846d with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:39.134737 14173 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:39.140028 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbef0147276d410da3d3830e682efe6d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.143486 13513 raft_consensus.cc:1050] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:39.144455 13513 consensus_queue.cc:237] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:39.146790 10358 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.147369 10358 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.157008 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.160598 10358 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.161248 13382 raft_consensus.cc:1050] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:39.162221 13382 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:39.162384 14166 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.163782 13740 raft_consensus.cc:1050] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:39.164589 13740 consensus_queue.cc:237] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:39.167796 12273 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.173084 10215 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.192760 10619 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
W20240501 08:12:39.208869 10557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.136:42153, user_credentials={real_user=slave}} blocked reactor thread for 48075.6us
I20240501 08:12:39.232702 12403 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.233723 14218 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.235885 10490 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.236960 14187 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.237690 10491 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.237982 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:39.238955 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:39.242331 14188 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.253172 12667 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.257696 10749 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.258436 10750 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.260380 12273 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.261349 14188 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240501 08:12:39.261608 10297 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 99451.4us
I20240501 08:12:39.262069 14188 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.262282 13497 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.262830 12873 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:39.268214 12667 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:39.268504 12273 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:39.273324 10295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.130:46605, user_credentials={real_user=slave}} blocked reactor thread for 112373us
I20240501 08:12:39.274170 13497 raft_consensus.cc:1050] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:39.274960 13497 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:39.278342 12273 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.278322 10220 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:39.279436 12873 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.279698 10752 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.280273 12873 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.281491 10748 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:39.284263 12873 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.287680 13953 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.288625 14177 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.289502 12667 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.290071 14172 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.291666 12665 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.292516 14173 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.293028 12665 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.293916 10748 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.294090 12273 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.300076 14172 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.301334 13986 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.306854 10358 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.306310 10752 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.308193 10622 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.308519 10492 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.315229 12405 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.321288 10212 catalog_manager.cc:5484] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.322841 14178 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.323530 14178 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.337087 10492 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.338038 10359 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:39.341171 10492 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.342173 14218 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.342147 10358 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.348040 10752 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.332877 10214 catalog_manager.cc:5484] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 reported cstate change: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
W20240501 08:12:39.354552 10558 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:39.356598 12535 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.357146 12534 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.357497 14173 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.358058 12535 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.359217 10492 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.359578 14186 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.361174 12403 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.361126 12405 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.364073 10490 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.369874 10750 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.371137 13740 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.371448 14171 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.372896 14172 consensus_queue.cc:1037] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.374861 14172 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.378816 10621 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.380872 13513 raft_consensus.cc:1013] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer 441e281fafbf42f8938c3f2f42ff7527: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:12:39.382946 14172 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.384238 14186 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.390331 14178 raft_consensus.cc:1013] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: attempt to promote peer 441e281fafbf42f8938c3f2f42ff7527: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:12:39.390861 14174 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.390915 10489 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.391068 10750 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.395641 12534 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.397432 10621 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.400260 12873 raft_consensus.cc:1050] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:39.401213 12873 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:39.402118 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d188d415f63f448195ae90e8e3d66e0e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.403463 10215 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.404990 12534 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.405906 10621 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.406409 14184 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.407045 10750 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.407516 10490 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.412930 13382 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.415208 14187 raft_consensus.cc:1050] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:39.416042 14187 consensus_queue.cc:237] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:39.419790 12873 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.420370 10619 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:39.422113 12405 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.423980 10212 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:39.428247 13382 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.428717 13382 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.433663 10358 consensus_queue.cc:237] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:39.434329 10359 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:39.435289 14177 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.437763 10749 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.437875 12665 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.438410 14171 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.438881 12667 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.438838 14171 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.439682 13969 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.440028 12535 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.441547 12273 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.442076 12665 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.443235 13969 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.443830 10752 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.447785 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae946c165cd34853917e5c8a244b6408 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.448058 12666 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.449687 14186 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.416271 14168 ts_tablet_manager.cc:1888] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.451604 14168 log.cc:1196] T b72323b8b99f4fd1aedcce69089f6204 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/b72323b8b99f4fd1aedcce69089f6204
I20240501 08:12:39.453258 13382 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.455158 10218 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.457224 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet b72323b8b99f4fd1aedcce69089f6204 (table rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]) successfully deleted
I20240501 08:12:39.451767 12273 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.458613 10621 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.459450 10359 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.464959 14188 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.467934 10619 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.471299 12403 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.474776 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.489650 10751 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.467558 13740 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.497735 12534 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.498502 10749 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.498652 14189 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.498354 12667 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.499037 13513 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.500702 14189 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.500983 12405 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.500975 12272 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.503275 14173 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.508884 10215 catalog_manager.cc:5484] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.509474 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:39.511816 10492 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.513119 12516 tablet_service.cc:1514] Processing DeleteTablet for tablet d188d415f63f448195ae90e8e3d66e0e with delete_type TABLET_DATA_TOMBSTONED (TS 441e281fafbf42f8938c3f2f42ff7527 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:32994
I20240501 08:12:39.513924 10219 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.513787 10487 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.518023 10490 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.518947 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet ae946c165cd34853917e5c8a244b6408 with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:39.520584 14168 tablet_replica.cc:330] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:39.521723 14168 raft_consensus.cc:2229] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.524288 14168 raft_consensus.cc:2258] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.521149 14189 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 08:12:39.534154 14168 ts_tablet_manager.cc:1875] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.544329 10212 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 5 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:39.549515 10358 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:39.554273 10358 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:39.557740 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:39.558463 14269 tablet_replica.cc:330] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: stopping tablet replica
I20240501 08:12:39.559408 14269 raft_consensus.cc:2229] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:39.559782 14269 raft_consensus.cc:2258] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:39.560947 14168 ts_tablet_manager.cc:1888] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:39.561884 14168 log.cc:1196] T ae946c165cd34853917e5c8a244b6408 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae946c165cd34853917e5c8a244b6408
I20240501 08:12:39.562947 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet ae946c165cd34853917e5c8a244b6408 (table rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]) successfully deleted
I20240501 08:12:39.565932 10358 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:39.566448 10491 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.568229 10749 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:39.568938 14186 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:39.576385 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:39.578398 10749 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.585775 14187 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.588892 10219 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
W20240501 08:12:39.591601 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.129:44049, user_credentials={real_user=slave}} blocked reactor thread for 82622.2us
I20240501 08:12:39.593919 12665 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.595149 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet bbef0147276d410da3d3830e682efe6d with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:39.597105 12402 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.597680 12401 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.598184 14186 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.600824 14269 ts_tablet_manager.cc:1875] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.601966 14177 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.603238 14177 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.604977 14186 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.606217 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.607728 10218 catalog_manager.cc:5484] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.609133 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.611646 10220 catalog_manager.cc:5484] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:39.615309 12404 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.620873 12534 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.621340 12664 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:39.622267 12873 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:39.622710 10731 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:39.622937 12873 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.624781 12404 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.625633 14177 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.629557 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae946c165cd34853917e5c8a244b6408 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.631793 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 92153666aebe47ddb080278cf24d1570 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.634058 10358 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.639052 10487 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.639417 14172 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.640411 10750 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.642143 10487 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.645466 10217 catalog_manager.cc:5484] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 reported cstate change: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.646338 10212 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 6 to 7, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.646684 10491 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.649418 14172 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.650089 10487 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.653129 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d188d415f63f448195ae90e8e3d66e0e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.654678 10732 tablet_service.cc:1514] Processing DeleteTablet for tablet 92153666aebe47ddb080278cf24d1570 with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:39.655710 14186 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.655652 12403 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.657330 12873 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.657886 10487 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.659090 12534 raft_consensus.cc:1203] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:39.661636 10488 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:39.667135 10490 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.671532 14273 tablet_replica.cc:330] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:39.671738 14273 raft_consensus.cc:2229] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.671974 14273 raft_consensus.cc:2258] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.672642 14272 tablet_replica.cc:330] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:39.672840 14272 raft_consensus.cc:2229] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.673079 14272 raft_consensus.cc:2258] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.672606 10490 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.675983 12666 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.676931 10220 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:39.680557 14272 ts_tablet_manager.cc:1875] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.681643 14273 ts_tablet_manager.cc:1875] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.679610 12535 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.697721 14272 ts_tablet_manager.cc:1888] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.697927 14272 log.cc:1196] T bbef0147276d410da3d3830e682efe6d P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/bbef0147276d410da3d3830e682efe6d
I20240501 08:12:39.700376 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet bbef0147276d410da3d3830e682efe6d (table rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]) successfully deleted
I20240501 08:12:39.633796 10751 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.712193 10215 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.713203 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:39.715102 13382 raft_consensus.cc:1050] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:39.715974 13382 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:39.718190 14273 ts_tablet_manager.cc:1888] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.718986 10488 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.720420 14187 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.722676 12535 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.723177 14187 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.720475 12664 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.732394 14188 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.732149 13382 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.734078 12664 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.740562 12534 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.742532 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.756451 10488 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.768661 12254 tablet_service.cc:1514] Processing DeleteTablet for tablet ae946c165cd34853917e5c8a244b6408 with delete_type TABLET_DATA_TOMBSTONED (TS bb2f0099fb794400b5c24927e213e5d8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53270
I20240501 08:12:39.772820 10621 consensus_queue.cc:237] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:39.776747 10619 consensus_queue.cc:237] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:39.781680 10359 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.782212 14189 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.782320 12403 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.782827 14189 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:12:39.783030 10205 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 6624441a1f1143648913595f905f8039 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240501 08:12:39.786983 14273 log.cc:1196] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/6e56f5f3cd9f4a56a3d43543ac0e5d14
I20240501 08:12:39.786901 14221 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.790495 10491 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.791429 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea502ae00eff4f72a74b8d25f5a11ede with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.791491 10491 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.792239 10487 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.793413 12404 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.794842 14273 tablet_replica.cc:330] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:39.795025 14273 raft_consensus.cc:2229] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.797346 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 (table rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]) successfully deleted
I20240501 08:12:39.798789 10212 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:39.799855 14273 raft_consensus.cc:2258] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.801120 13513 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.801887 13513 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.803259 14273 ts_tablet_manager.cc:1875] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.804205 10491 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.805018 12273 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.807045 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.807260 14187 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.809581 14189 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.811993 10490 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.815295 12253 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.816100 10358 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.822089 12272 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.829708 14287 tablet_replica.cc:330] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: stopping tablet replica
I20240501 08:12:39.815295 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b6f09b0b5e84329944360d511578ed0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.839686 10219 catalog_manager.cc:5484] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:39.757939 10731 tablet_service.cc:1514] Processing DeleteTablet for tablet d188d415f63f448195ae90e8e3d66e0e with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:39.778628 10751 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:39.733558 10752 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:39.848168 14187 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:39.850272 10752 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.852468 14287 raft_consensus.cc:2229] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.853643 14287 raft_consensus.cc:2258] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.845089 10214 catalog_manager.cc:5484] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
W20240501 08:12:39.847046 10205 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 9b6f09b0b5e84329944360d511578ed0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240501 08:12:39.859858 10619 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:39.864251 10622 consensus_queue.cc:237] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:39.867745 12536 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.867739 10358 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.868402 14166 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.868520 13382 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.869554 10752 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.870149 13382 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.870425 12534 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.871102 14218 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.871613 14218 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.872845 14166 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.875612 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa6139020ac7458aa8a7ac08f63e6f26 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.877871 12536 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.878787 10752 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.880532 14287 ts_tablet_manager.cc:1875] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.880658 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.880988 10359 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.881619 13856 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:39.882350 13856 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:39.883004 10751 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.883579 12273 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.884465 14187 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.887379 14166 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.890460 13382 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.892632 13856 ts_tablet_manager.cc:1367] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 1.616s	user 0.053s	sys 0.008s
I20240501 08:12:39.897518 12666 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.898134 14189 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.900566 12666 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.901307 12273 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.902622 10731 tablet_service.cc:1514] Processing DeleteTablet for tablet ea502ae00eff4f72a74b8d25f5a11ede with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:39.903439 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:39.905835 10358 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:39.908896 12665 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.909404 14186 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.910208 10487 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.910831 14186 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.913388 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6624441a1f1143648913595f905f8039 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.915179 14186 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.917920 13856 raft_consensus.cc:348] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:39.918251 13856 raft_consensus.cc:729] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:39.918525 13856 consensus_queue.cc:260] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:39.919271 13856 ts_tablet_manager.cc:1398] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:39.919728 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 850d4e1020e243f4b6fa113d2946846d with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.919974 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-cd01dd912e1e4ca9a104270f3cd004b1 received from {username='slave'} at 127.2.248.134:51519
I20240501 08:12:39.920106 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-cd01dd912e1e4ca9a104270f3cd004b1 on tablet cd01dd912e1e4ca9a104270f3cd004b1 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:39.921093 10488 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.922282 12665 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.925676 10751 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.928225 10219 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.930315 14273 ts_tablet_manager.cc:1888] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:12:39.933127 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:39.934573 10221 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.937394 10470 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:39.940723 10622 consensus_queue.cc:237] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:39.943303 12272 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.945216 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet 850d4e1020e243f4b6fa113d2946846d with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:39.945518 14168 tablet_replica.cc:330] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:39.945672 14168 raft_consensus.cc:2229] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.945974 14168 raft_consensus.cc:2258] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.947000 14168 ts_tablet_manager.cc:1875] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.947783 14187 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.958472 10487 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:39.959062 13513 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:39.959501 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.959683 14188 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.959901 12272 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.963557 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b6f09b0b5e84329944360d511578ed0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.964442 14168 ts_tablet_manager.cc:1888] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:39.964548 14168 log.cc:1196] T 850d4e1020e243f4b6fa113d2946846d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/850d4e1020e243f4b6fa113d2946846d
I20240501 08:12:39.965261 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet 850d4e1020e243f4b6fa113d2946846d (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:39.966305 10491 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.971940 10217 catalog_manager.cc:5484] T 9b6f09b0b5e84329944360d511578ed0 P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:39.984728 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6f09b0b5e84329944360d511578ed0 with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:39.988677 10221 catalog_manager.cc:5484] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:39.989248 10221 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:39.994124 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6139020ac7458aa8a7ac08f63e6f26 with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:39.994263 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 6624441a1f1143648913595f905f8039 with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:39.994535 14272 tablet_replica.cc:330] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:39.994730 14272 raft_consensus.cc:2229] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.994943 14272 raft_consensus.cc:2258] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.995895 12516 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6f09b0b5e84329944360d511578ed0 with delete_type TABLET_DATA_TOMBSTONED (TS 441e281fafbf42f8938c3f2f42ff7527 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:32994
I20240501 08:12:39.996939 14272 ts_tablet_manager.cc:1875] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.997954 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:39.999800 12536 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:40.000448 14186 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:40.001746 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.002094 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:40.007086 14269 ts_tablet_manager.cc:1888] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:12:40.007318 12536 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.018337 14306 tablet_replica.cc:330] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:40.022966 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.026809 14306 raft_consensus.cc:2229] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.027266 14306 raft_consensus.cc:2258] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.032322 14272 ts_tablet_manager.cc:1888] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:40.032482 14272 log.cc:1196] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa6139020ac7458aa8a7ac08f63e6f26
I20240501 08:12:40.033582 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet aa6139020ac7458aa8a7ac08f63e6f26 (table rebalance_test_table_4 [id=dc637c58564a476c8f56248349c3625e]) successfully deleted
I20240501 08:12:40.035614 10619 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:40.039443 14306 ts_tablet_manager.cc:1875] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.041119 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:40.004492 12534 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:40.041440 14168 tablet_replica.cc:330] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:40.041639 14168 raft_consensus.cc:2229] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.041870 14168 raft_consensus.cc:2258] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.042728 14168 ts_tablet_manager.cc:1875] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.052996 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.053050 14168 ts_tablet_manager.cc:1888] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:40.053208 14168 log.cc:1196] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a6ed4ea6d6941d2af0f81eaf744bcea
I20240501 08:12:40.054503 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet 8a6ed4ea6d6941d2af0f81eaf744bcea (table rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]) successfully deleted
I20240501 08:12:40.058081 14269 log.cc:1196] T d188d415f63f448195ae90e8e3d66e0e P 441e281fafbf42f8938c3f2f42ff7527: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/wals/d188d415f63f448195ae90e8e3d66e0e
I20240501 08:12:40.068305 14272 tablet_replica.cc:330] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:40.068619 14272 raft_consensus.cc:2229] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.069234 14272 raft_consensus.cc:2258] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.069886 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.057847 10359 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:40.074218 14272 ts_tablet_manager.cc:1875] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.074759 10491 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:40.075420 14173 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:40.077755 14173 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:40.080328 10208 catalog_manager.cc:4830] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): tablet d188d415f63f448195ae90e8e3d66e0e (table rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]) successfully deleted
W20240501 08:12:40.080606 10208 catalog_manager.cc:4810] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): delete failed for tablet 9b6f09b0b5e84329944360d511578ed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b6f09b0b5e84329944360d511578ed0 already in progress: copying tablet
I20240501 08:12:40.080600 12666 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:40.081383 12873 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:40.082494 10491 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:40.085309 12665 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:40.085820 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 850d4e1020e243f4b6fa113d2946846d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:40.093830 12535 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:40.094677 14189 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:40.097249 14187 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.098086 12535 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.101365 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6624441a1f1143648913595f905f8039 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:40.104820 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.110021 10219 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:40.117043 12516 tablet_service.cc:1514] Processing DeleteTablet for tablet 850d4e1020e243f4b6fa113d2946846d with delete_type TABLET_DATA_TOMBSTONED (TS 441e281fafbf42f8938c3f2f42ff7527 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:32994
I20240501 08:12:40.126956 14269 tablet_replica.cc:330] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: stopping tablet replica
I20240501 08:12:40.127242 14269 raft_consensus.cc:2229] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.127779 14269 raft_consensus.cc:2258] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.129169 14269 ts_tablet_manager.cc:1875] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.143047 14306 ts_tablet_manager.cc:1888] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:40.143218 14306 log.cc:1196] T 9b6f09b0b5e84329944360d511578ed0 P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b6f09b0b5e84329944360d511578ed0
I20240501 08:12:40.151361 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet 9b6f09b0b5e84329944360d511578ed0 (table rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]) successfully deleted
I20240501 08:12:40.156239 14272 ts_tablet_manager.cc:1888] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:40.156505 14272 log.cc:1196] T 6624441a1f1143648913595f905f8039 P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/6624441a1f1143648913595f905f8039
I20240501 08:12:40.157428 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet 6624441a1f1143648913595f905f8039 (table rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]) successfully deleted
I20240501 08:12:40.170076 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:40.184175 10220 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 8 to 9, VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:40.212242 14273 log.cc:1196] T 92153666aebe47ddb080278cf24d1570 P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/92153666aebe47ddb080278cf24d1570
I20240501 08:12:40.213348 14273 tablet_replica.cc:330] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:40.213613 14273 raft_consensus.cc:2229] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.213908 14273 raft_consensus.cc:2258] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.974404 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:40.038573 10751 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:40.216234 14189 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:40.216337 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet 92153666aebe47ddb080278cf24d1570 (table rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]) successfully deleted
I20240501 08:12:40.217736 10751 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.240600 14287 ts_tablet_manager.cc:1888] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:12:40.240845 14287 log.cc:1196] T ae946c165cd34853917e5c8a244b6408 P bb2f0099fb794400b5c24927e213e5d8: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/wals/ae946c165cd34853917e5c8a244b6408
I20240501 08:12:40.243420 10207 catalog_manager.cc:4830] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): tablet ae946c165cd34853917e5c8a244b6408 (table rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]) successfully deleted
I20240501 08:12:40.244498 14273 ts_tablet_manager.cc:1875] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.263015 14269 ts_tablet_manager.cc:1888] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:12:40.267658 12647 tablet_service.cc:1514] Processing DeleteTablet for tablet 6624441a1f1143648913595f905f8039 with delete_type TABLET_DATA_TOMBSTONED (TS c9f609481ded4d1aad9dd5f2341fd931 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60706
I20240501 08:12:40.291141 14273 ts_tablet_manager.cc:1888] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:40.298377 12403 raft_consensus.cc:1203] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:40.304678 14318 raft_consensus.cc:1050] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:40.305476 14318 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:40.328177 10359 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:40.329041 14270 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:40.330993 14319 tablet_replica.cc:330] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: stopping tablet replica
I20240501 08:12:40.331491 14319 raft_consensus.cc:2229] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.332026 14319 raft_consensus.cc:2258] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.333957 12534 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:40.334834 14315 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:40.349864 14269 log.cc:1196] T 850d4e1020e243f4b6fa113d2946846d P 441e281fafbf42f8938c3f2f42ff7527: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/wals/850d4e1020e243f4b6fa113d2946846d
I20240501 08:12:40.363307 10208 catalog_manager.cc:4830] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): tablet 850d4e1020e243f4b6fa113d2946846d (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:40.378427 10359 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:40.381337 12665 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 08:12:40.381500 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 850d4e1020e243f4b6fa113d2946846d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.382292 10488 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:40.382934 14173 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:40.383426 14173 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:40.388290 12873 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:40.393849 12665 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:40.396392 10488 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:40.405651 10219 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:40.419692 14319 ts_tablet_manager.cc:1875] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.329444 10751 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:40.426740 14273 log.cc:1196] T d188d415f63f448195ae90e8e3d66e0e P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/d188d415f63f448195ae90e8e3d66e0e
I20240501 08:12:40.427261 14317 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:40.428416 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet d188d415f63f448195ae90e8e3d66e0e (table rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]) successfully deleted
I20240501 08:12:40.433949 14273 tablet_replica.cc:330] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:40.434379 14273 raft_consensus.cc:2229] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.438192 14273 raft_consensus.cc:2258] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.438185 14320 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.438668 12535 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.443650 10359 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.444556 10218 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 reported cstate change: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.448827 14273 ts_tablet_manager.cc:1875] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.473843 10752 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.475153 14017 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:40.475597 14017 tablet_bootstrap.cc:492] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:40.475865 14017 ts_tablet_manager.cc:1367] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 1.984s	user 0.052s	sys 0.011s
I20240501 08:12:40.476670 14017 raft_consensus.cc:348] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:40.483981 14017 raft_consensus.cc:729] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:40.484274 14017 consensus_queue.cc:260] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:40.484983 14017 ts_tablet_manager.cc:1398] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I20240501 08:12:40.484548 10488 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:40.486877 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-9b6f09b0b5e84329944360d511578ed0 received from {username='slave'} at 127.2.248.135:55689
I20240501 08:12:40.487005 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-9b6f09b0b5e84329944360d511578ed0 on tablet 9b6f09b0b5e84329944360d511578ed0 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:40.487649 10752 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:40.487712 12516 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6f09b0b5e84329944360d511578ed0 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:32994
I20240501 08:12:40.488561 14317 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:40.490692 14269 tablet_replica.cc:330] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: stopping tablet replica
I20240501 08:12:40.490897 14269 raft_consensus.cc:2229] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:40.491079 14269 raft_consensus.cc:2258] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:40.491801 14269 ts_tablet_manager.cc:1875] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.494307 12534 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:40.496376 14317 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:40.513062 14320 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.514209 12535 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.517241 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 185a62881ea84131a0c14987e4296338 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:40.526238 10218 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 reported cstate change: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.529507 14319 ts_tablet_manager.cc:1888] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:12:40.534564 14319 log.cc:1196] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-7/wal/wals/6624441a1f1143648913595f905f8039
I20240501 08:12:40.535845 10207 catalog_manager.cc:4830] TS c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): tablet 6624441a1f1143648913595f905f8039 (table rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]) successfully deleted
I20240501 08:12:40.540283 10752 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:40.543547 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet 185a62881ea84131a0c14987e4296338 with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:40.543915 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.543952 14306 tablet_replica.cc:330] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:40.544196 14306 raft_consensus.cc:2229] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.544479 14306 raft_consensus.cc:2258] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.547739 14306 ts_tablet_manager.cc:1875] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.551290 14273 ts_tablet_manager.cc:1888] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:40.551434 14273 log.cc:1196] T ea502ae00eff4f72a74b8d25f5a11ede P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/ea502ae00eff4f72a74b8d25f5a11ede
I20240501 08:12:40.552160 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet ea502ae00eff4f72a74b8d25f5a11ede (table rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]) successfully deleted
I20240501 08:12:40.559854 14306 ts_tablet_manager.cc:1888] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:40.560077 14306 log.cc:1196] T 185a62881ea84131a0c14987e4296338 P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/185a62881ea84131a0c14987e4296338
I20240501 08:12:40.561074 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet 185a62881ea84131a0c14987e4296338 (table rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]) successfully deleted
I20240501 08:12:40.561386 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:40.583122 13968 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:40.597178 13968 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:40.604439 13968 ts_tablet_manager.cc:1367] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 2.351s	user 0.052s	sys 0.014s
I20240501 08:12:40.607295 13968 raft_consensus.cc:348] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:40.608211 13968 raft_consensus.cc:729] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:40.608584 13968 consensus_queue.cc:260] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:40.615020 13968 ts_tablet_manager.cc:1398] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240501 08:12:40.615888 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-92153666aebe47ddb080278cf24d1570 received from {username='slave'} at 127.2.248.135:55933
I20240501 08:12:40.616014 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-92153666aebe47ddb080278cf24d1570 on tablet 92153666aebe47ddb080278cf24d1570 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:40.623485 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:40.639623 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:40.677702 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.680797 14269 ts_tablet_manager.cc:1888] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:40.681113 14269 log.cc:1196] T 9b6f09b0b5e84329944360d511578ed0 P 441e281fafbf42f8938c3f2f42ff7527: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-6/wal/wals/9b6f09b0b5e84329944360d511578ed0
I20240501 08:12:40.692623 14166 raft_consensus.cc:1050] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:40.693598 14166 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:40.772678 10208 catalog_manager.cc:4830] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): tablet 9b6f09b0b5e84329944360d511578ed0 (table rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]) successfully deleted
I20240501 08:12:40.773756 10622 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:40.774461 12403 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:40.776489 14363 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:40.781666 14166 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:40.792716 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.800804 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:40.818919 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:12:40.819448 10687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.133:43981, user_credentials={real_user=slave}} blocked reactor thread for 52042.7us
I20240501 08:12:40.820439 10359 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:40.820747 14360 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:40.822428 12404 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:40.825199 14363 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:40.830982 10622 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:40.843741 14362 ts_tablet_manager.cc:902] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:40.834542 10218 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:40.875912 14362 tablet_copy_client.cc:308] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:40.878423 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 850d4e1020e243f4b6fa113d2946846d from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:60051)
I20240501 08:12:40.878629 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 850d4e1020e243f4b6fa113d2946846d from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:60051: session id = bb2f0099fb794400b5c24927e213e5d8-850d4e1020e243f4b6fa113d2946846d
I20240501 08:12:40.883168 10381 tablet_copy_source_session.cc:205] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:40.885052 14362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d4e1020e243f4b6fa113d2946846d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.890651 10358 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:40.899474 14362 tablet_copy_client.cc:783] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:40.899735 14362 tablet_copy_client.cc:647] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:40.905606 14362 tablet_copy_client.cc:523] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:40.911074 12534 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:40.911809 12873 consensus_queue.cc:1037] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:40.914502 12534 raft_consensus.cc:858] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:40.914652 12534 raft_consensus.cc:1565] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:40.926676 14362 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:40.926604 12534 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:40.933871 14186 raft_consensus.cc:1050] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:40.934690 14186 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:40.950033 12534 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:40.958562 12404 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:40.960281 10488 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:40.966038 12873 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:40.983031 12873 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:40.996788 12534 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:40.997541 14375 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:41.005745 14186 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.006358 10491 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.016793 10218 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.020560 12404 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.032012 12534 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.049124 10358 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:41.053040 12535 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:41.053123 12404 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:41.053822 14377 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:41.054081 12873 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:41.058481 14377 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.065433 12535 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.072136 10622 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:41.071956 12404 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.072052 10219 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P 441e281fafbf42f8938c3f2f42ff7527 reported cstate change: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:41.078246 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 92153666aebe47ddb080278cf24d1570 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:41.079531 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 92153666aebe47ddb080278cf24d1570 with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44486
W20240501 08:12:41.094552 10555 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b924c6344714a92ac37ae28961b02f5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.095775 10752 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.096467 13382 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.097323 10491 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.097398 14397 tablet_replica.cc:330] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:41.097810 14397 raft_consensus.cc:2229] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:41.098284 14397 raft_consensus.cc:2258] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:41.098960 13382 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.100322 14397 ts_tablet_manager.cc:1875] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:41.117468 13382 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:41.120136 14397 ts_tablet_manager.cc:1888] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:12:41.120378 14397 log.cc:1196] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/92153666aebe47ddb080278cf24d1570
I20240501 08:12:41.121568 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet 92153666aebe47ddb080278cf24d1570 (table rebalance_test_table_2 [id=c4a9863b0a3c4aa0b5ab510cebd81045]) successfully deleted
I20240501 08:12:41.125227 10218 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.125571 10491 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:41.139623 10622 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:41.147643 14362 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240501 08:12:41.149251 10557 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b924c6344714a92ac37ae28961b02f5. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:41.149591 10555 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b924c6344714a92ac37ae28961b02f5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.153236 14362 tablet_bootstrap.cc:492] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:41.153599 14362 ts_tablet_manager.cc:1367] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.227s	user 0.073s	sys 0.027s
I20240501 08:12:41.156972 10491 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.158665 10751 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.159123 13382 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.159628 14189 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.156245 14362 raft_consensus.cc:348] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:41.161180 10751 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:41.161993 14362 raft_consensus.cc:729] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:41.162282 14362 consensus_queue.cc:260] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:41.177160 14189 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:41.180891 10488 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:41.188588 10752 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:41.190789 10218 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.201949 14362 ts_tablet_manager.cc:1398] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.048s	user 0.013s	sys 0.006s
I20240501 08:12:41.202989 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-850d4e1020e243f4b6fa113d2946846d received from {username='slave'} at 127.2.248.133:60051
I20240501 08:12:41.203119 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-850d4e1020e243f4b6fa113d2946846d on tablet 850d4e1020e243f4b6fa113d2946846d with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:41.248541 10752 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:41.252080 12404 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:41.259516 13953 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:41.261837 12273 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:41.262670 10622 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:41.265738 14166 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:41.266408 14166 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:41.290040 10752 consensus_queue.cc:237] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
W20240501 08:12:41.303362 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a63e16f11894c2886091dcd3d204106. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.304337 10358 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.304946 13953 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.312053 10488 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.313581 14166 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.330407 14364 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:41.336014 12272 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:41.336819 10619 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:41.342439 10219 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 10 to 11. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.345059 12403 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:41.351910 10752 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } }
I20240501 08:12:41.353765 14166 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:41.354408 10491 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:41.359053 10361 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:41.360338 10622 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:41.360934 14364 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:12:41.362738 10219 catalog_manager.cc:5484] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:41.363896 12404 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:41.364255 12272 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:41.365286 13953 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 08:12:41.366659 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.370898 14363 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:12:41.375895 13953 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.376487 12404 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.382910 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af24b8ab4d0c43ef986d89d4aefc263c with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:12:41.384265 10619 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.389358 10221 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 11 to 12, NON_VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.403431 12272 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.416379 10752 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } }
I20240501 08:12:41.419875 12403 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:12:41.421435 14166 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:12:41.423899 12272 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:12:41.425383 14364 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.432013 12403 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.438825 14363 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:12:41.440800 10220 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 12 to 13, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.442162 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af24b8ab4d0c43ef986d89d4aefc263c with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:41.443467 12272 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } } }
I20240501 08:12:41.446417 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet af24b8ab4d0c43ef986d89d4aefc263c with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:40014
W20240501 08:12:41.448444 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.449970 10361 consensus_queue.cc:237] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:41.464660 10752 raft_consensus.cc:1261] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.465543 14377 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:41.467545 10295 consensus_peers.cc:485] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5a58b7166534844b0963f4fab46dfd9. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.473073 10619 raft_consensus.cc:1261] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.473887 14377 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.476315 14459 tablet_replica.cc:330] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:41.476644 14459 raft_consensus.cc:2229] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:41.476958 14459 raft_consensus.cc:2258] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:41.478466 14459 ts_tablet_manager.cc:1875] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:41.492038 14377 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:41.495016 14459 ts_tablet_manager.cc:1888] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 08:12:41.495251 14459 log.cc:1196] T af24b8ab4d0c43ef986d89d4aefc263c P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/af24b8ab4d0c43ef986d89d4aefc263c
I20240501 08:12:41.496450 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet af24b8ab4d0c43ef986d89d4aefc263c (table rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]) successfully deleted
I20240501 08:12:41.504310 10619 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:41.516381 10219 catalog_manager.cc:5484] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:41.526593 14462 log.cc:924] Time spent T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240501 08:12:41.531824 10752 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:41.576907 10488 consensus_queue.cc:237] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:41.585402 10487 consensus_queue.cc:237] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:41.589915 10622 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.591020 10361 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:41.591624 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3453d8d883014219b1b14c88e250736b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.592306 14320 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.596926 14320 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.622123 14320 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:41.665813 12272 raft_consensus.cc:1203] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20240501 08:12:41.671299 10427 consensus_peers.cc:485] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4799a93d96a64a75b1984bb85fcad7e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.675017 10752 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.675904 14486 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:41.682619 10425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.131:34319, user_credentials={real_user=slave}} blocked reactor thread for 55290.7us
I20240501 08:12:41.684145 10622 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.689216 14489 ts_tablet_manager.cc:902] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:41.691044 14362 ts_tablet_manager.cc:902] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:41.702253 10361 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:41.703372 10221 catalog_manager.cc:5484] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.722285 14485 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.722443 10622 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:41.741171 14492 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:41.745180 10752 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:41.766729 14507 ts_tablet_manager.cc:902] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:41.766893 10619 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:41.782850 10221 catalog_manager.cc:5484] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:41.800597 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:41.828884 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:41.849850 14531 ts_tablet_manager.cc:902] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:41.862326 14507 tablet_copy_client.cc:308] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:41.865783 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:41.874253 10622 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:41.876251 14489 tablet_copy_client.cc:308] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:41.877033 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 2a63e16f11894c2886091dcd3d204106 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47327)
I20240501 08:12:41.877205 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 2a63e16f11894c2886091dcd3d204106 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47327: session id = bb2f0099fb794400b5c24927e213e5d8-2a63e16f11894c2886091dcd3d204106
I20240501 08:12:41.878983 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 4b924c6344714a92ac37ae28961b02f5 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:42583)
I20240501 08:12:41.879158 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 4b924c6344714a92ac37ae28961b02f5 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:42583: session id = b6a964e54f3b44eda6d0666e0c1e4bea-4b924c6344714a92ac37ae28961b02f5
I20240501 08:12:41.880074 10772 tablet_copy_source_session.cc:205] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.882117 10642 tablet_copy_source_session.cc:205] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.887838 14362 tablet_copy_client.cc:308] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
W20240501 08:12:41.896787 10557 consensus_peers.cc:485] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.897230 10752 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:41.897637 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 4b924c6344714a92ac37ae28961b02f5 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47709)
I20240501 08:12:41.898372 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 4b924c6344714a92ac37ae28961b02f5 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47709: session id = bb2f0099fb794400b5c24927e213e5d8-4b924c6344714a92ac37ae28961b02f5
I20240501 08:12:41.902638 10642 tablet_copy_source_session.cc:205] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.904489 14507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a63e16f11894c2886091dcd3d204106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.905685 12403 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:41.907239 14189 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:41.907541 14489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924c6344714a92ac37ae28961b02f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.916553 14362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b924c6344714a92ac37ae28961b02f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.917727 13382 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:41.935653 14362 tablet_copy_client.cc:783] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.936146 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:41.937106 14362 tablet_copy_client.cc:647] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.946769 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:41.949146 14189 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:41.964048 10752 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:41.971933 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:41.981638 14362 tablet_copy_client.cc:523] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.981932 12403 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:41.985358 14362 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:41.991508 10219 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.992961 14531 tablet_copy_client.cc:308] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:41.994390 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet af24b8ab4d0c43ef986d89d4aefc263c from peer c3747ec2a93c4684962f215616e896ed ({username='slave'} at 127.2.248.129:40069)
I20240501 08:12:41.994571 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet af24b8ab4d0c43ef986d89d4aefc263c from peer c3747ec2a93c4684962f215616e896ed at {username='slave'} at 127.2.248.129:40069: session id = c3747ec2a93c4684962f215616e896ed-af24b8ab4d0c43ef986d89d4aefc263c
I20240501 08:12:41.997601 10772 tablet_copy_source_session.cc:205] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.999157 14531 ts_tablet_manager.cc:1875] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:12:42.009323 14531 ts_tablet_manager.cc:1888] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 08:12:42.009685 14531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af24b8ab4d0c43ef986d89d4aefc263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.026183 14489 tablet_copy_client.cc:783] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.026486 14489 tablet_copy_client.cc:647] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.026860 14507 tablet_copy_client.cc:783] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.027113 14507 tablet_copy_client.cc:647] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.029430 14489 tablet_copy_client.cc:523] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.030242 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.044535 14507 tablet_copy_client.cc:523] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.048902 14489 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:42.055335 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:42.083029 14507 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:42.092464 14531 tablet_copy_client.cc:783] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.092787 14531 tablet_copy_client.cc:647] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.110340 14531 tablet_copy_client.cc:523] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.117733 14564 ts_tablet_manager.cc:902] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
W20240501 08:12:42.125989 10557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 87096.9us
I20240501 08:12:42.148905 14377 raft_consensus.cc:1050] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:42.151501 14377 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:42.159667 10488 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:42.161234 12272 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:42.161911 12665 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:42.164175 12873 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:42.164971 12873 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:42.165549 12873 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:42.176362 12873 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:42.176975 12665 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:42.190284 14362 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.190786 14362 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:42.190511 12272 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:42.191059 14362 ts_tablet_manager.cc:1367] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.206s	user 0.048s	sys 0.008s
I20240501 08:12:42.191931 14362 raft_consensus.cc:348] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:42.192260 14362 raft_consensus.cc:729] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:42.205258 10219 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 10 to 11, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:42.207470 14583 ts_tablet_manager.cc:902] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:42.211380 10361 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:42.217756 14489 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.218232 14489 tablet_bootstrap.cc:492] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:42.218550 14489 ts_tablet_manager.cc:1367] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.170s	user 0.041s	sys 0.017s
I20240501 08:12:42.219375 14489 raft_consensus.cc:348] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:42.219743 14489 raft_consensus.cc:729] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:42.219956 14489 consensus_queue.cc:260] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:42.227672 14489 ts_tablet_manager.cc:1398] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20240501 08:12:42.229866 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-4b924c6344714a92ac37ae28961b02f5 received from {username='slave'} at 127.2.248.134:42583
I20240501 08:12:42.229984 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-4b924c6344714a92ac37ae28961b02f5 on tablet 4b924c6344714a92ac37ae28961b02f5 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:42.233520 14531 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Bootstrap starting.
I20240501 08:12:42.245882 10752 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:42.260241 14578 ts_tablet_manager.cc:902] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:42.272171 10488 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:42.272758 14364 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:42.275871 14166 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:42.276501 10487 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:42.287863 14564 tablet_copy_client.cc:308] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:42.289549 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet f5a58b7166534844b0963f4fab46dfd9 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:46779)
I20240501 08:12:42.289760 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet f5a58b7166534844b0963f4fab46dfd9 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:46779: session id = b6a964e54f3b44eda6d0666e0c1e4bea-f5a58b7166534844b0963f4fab46dfd9
I20240501 08:12:42.294281 10381 tablet_copy_source_session.cc:205] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.296411 14564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a58b7166534844b0963f4fab46dfd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:42.306758 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50b7150bbeb645d1b00d1a561c3276d1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:42.318563 10220 catalog_manager.cc:5484] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:42.316085 10752 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:42.356470 12274 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:42.357187 14166 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:42.358894 14578 tablet_copy_client.cc:308] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:42.359845 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 3453d8d883014219b1b14c88e250736b from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:48525)
I20240501 08:12:42.360002 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 3453d8d883014219b1b14c88e250736b from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:48525: session id = 441e281fafbf42f8938c3f2f42ff7527-3453d8d883014219b1b14c88e250736b
I20240501 08:12:42.363219 10512 tablet_copy_source_session.cc:205] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.366091 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3453d8d883014219b1b14c88e250736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.365974 14362 consensus_queue.cc:260] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:42.369596 12666 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:42.371114 12873 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:12:42.375764 12873 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
W20240501 08:12:42.377408 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21065b7dcc18456eb1b4c2376d51eec4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:42.378387 10361 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.378939 14166 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.380585 12666 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:42.386862 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 850d4e1020e243f4b6fa113d2946846d with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.387959 10219 catalog_manager.cc:5484] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 11 to 12, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:42.390230 10622 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.390882 14364 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.400781 14166 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:42.403273 10622 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:42.407891 10218 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:42.435528 10358 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:42.451027 14362 ts_tablet_manager.cc:1398] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.252s	user 0.005s	sys 0.000s
I20240501 08:12:42.452128 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-4b924c6344714a92ac37ae28961b02f5 received from {username='slave'} at 127.2.248.133:47709
I20240501 08:12:42.452275 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-4b924c6344714a92ac37ae28961b02f5 on tablet 4b924c6344714a92ac37ae28961b02f5 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:42.452965 14507 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.453476 14507 tablet_bootstrap.cc:492] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:42.453801 14507 ts_tablet_manager.cc:1367] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.371s	user 0.047s	sys 0.007s
I20240501 08:12:42.454821 14507 raft_consensus.cc:348] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:42.455166 14507 raft_consensus.cc:729] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:42.455457 14507 consensus_queue.cc:260] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:42.461995 14507 ts_tablet_manager.cc:1398] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.001s
I20240501 08:12:42.462631 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-2a63e16f11894c2886091dcd3d204106 received from {username='slave'} at 127.2.248.133:47327
I20240501 08:12:42.462791 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-2a63e16f11894c2886091dcd3d204106 on tablet 2a63e16f11894c2886091dcd3d204106 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:42.463383 12273 raft_consensus.cc:1261] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:42.464254 12873 consensus_queue.cc:1035] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:12:42.465752 12273 raft_consensus.cc:2932] T 850d4e1020e243f4b6fa113d2946846d P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:42.470037 14631 ts_tablet_manager.cc:902] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:42.470583 12274 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:42.478914 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 850d4e1020e243f4b6fa113d2946846d with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:42.501019 12272 raft_consensus.cc:1203] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:42.503314 14583 tablet_copy_client.cc:308] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:42.504187 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 4799a93d96a64a75b1984bb85fcad7e8 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:37801)
I20240501 08:12:42.504411 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 4799a93d96a64a75b1984bb85fcad7e8 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:37801: session id = c9f609481ded4d1aad9dd5f2341fd931-4799a93d96a64a75b1984bb85fcad7e8
I20240501 08:12:42.510977 10512 tablet_copy_source_session.cc:205] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.513783 14583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4799a93d96a64a75b1984bb85fcad7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.517808 14634 tablet_replica.cc:330] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:42.525475 14583 tablet_copy_client.cc:783] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.525724 14583 tablet_copy_client.cc:647] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.527702 14583 tablet_copy_client.cc:523] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.528178 14564 tablet_copy_client.cc:783] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.528430 14564 tablet_copy_client.cc:647] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.531774 14583 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:42.534999 14631 tablet_copy_client.cc:308] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:42.535830 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet cf1f28d15c094fd98fac1b91d2785744 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56369)
I20240501 08:12:42.536032 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet cf1f28d15c094fd98fac1b91d2785744 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56369: session id = c9f609481ded4d1aad9dd5f2341fd931-cf1f28d15c094fd98fac1b91d2785744
I20240501 08:12:42.518258 10487 consensus_queue.cc:237] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:42.539980 14634 raft_consensus.cc:2229] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.540242 14634 pending_rounds.cc:62] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Trying to abort 1 pending ops.
I20240501 08:12:42.540352 14634 pending_rounds.cc:68] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Aborting op as it isn't in flight: id { term: 1 index: 11 } timestamp: 7022801560163307520 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "850d4e1020e243f4b6fa113d2946846d" old_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } } new_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } } }
I20240501 08:12:42.541165 14634 raft_consensus.cc:2894] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Aborting config change with OpId 1.11: Aborted: Op aborted
I20240501 08:12:42.541438 14634 raft_consensus.cc:2866] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:12:42.541690 14634 raft_consensus.cc:2258] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.542560 10622 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:42.543597 14486 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:42.544230 14564 tablet_copy_client.cc:523] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:12:42.541234 10427 consensus_peers.cc:485] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4799a93d96a64a75b1984bb85fcad7e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:42.545720 10752 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:42.548332 14634 ts_tablet_manager.cc:1875] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:12:42.548409 10427 consensus_peers.cc:485] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:42.549604 14492 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:42.549301 14650 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:42.550019 10622 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:42.551133 10752 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:42.552855 10642 tablet_copy_source_session.cc:205] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.555938 14631 ts_tablet_manager.cc:1875] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:12:42.562337 14578 tablet_copy_client.cc:783] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.562601 14578 tablet_copy_client.cc:647] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.565935 14631 ts_tablet_manager.cc:1888] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:12:42.566308 14631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1f28d15c094fd98fac1b91d2785744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.577082 10215 catalog_manager.cc:5484] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:42.579826 14631 tablet_copy_client.cc:783] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.580292 14631 tablet_copy_client.cc:647] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.584625 14631 tablet_copy_client.cc:523] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.590638 14578 tablet_copy_client.cc:523] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.598945 14631 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:42.603116 12274 raft_consensus.cc:1203] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:42.652314 14564 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:42.663930 14578 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:42.681445 14583 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.682117 14583 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:42.682451 14583 ts_tablet_manager.cc:1367] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.151s	user 0.061s	sys 0.029s
I20240501 08:12:42.683562 14583 raft_consensus.cc:348] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:42.683902 14583 raft_consensus.cc:729] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:42.684882 14583 consensus_queue.cc:260] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:42.686503 14583 ts_tablet_manager.cc:1398] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240501 08:12:42.686676 14634 ts_tablet_manager.cc:1888] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 08:12:42.686861 14634 log.cc:1196] T 850d4e1020e243f4b6fa113d2946846d P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/850d4e1020e243f4b6fa113d2946846d
I20240501 08:12:42.703246 14666 ts_tablet_manager.cc:902] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:42.728169 12403 raft_consensus.cc:1203] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240501 08:12:42.687196 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-4799a93d96a64a75b1984bb85fcad7e8 received from {username='slave'} at 127.2.248.136:37801
I20240501 08:12:42.739212 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-4799a93d96a64a75b1984bb85fcad7e8 on tablet 4799a93d96a64a75b1984bb85fcad7e8 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:42.759122 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet 850d4e1020e243f4b6fa113d2946846d (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:42.767526 14666 tablet_copy_client.cc:308] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:42.797714 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:42.798126 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:42.805716 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.839401 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 50b7150bbeb645d1b00d1a561c3276d1 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:44587)
I20240501 08:12:42.839594 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 50b7150bbeb645d1b00d1a561c3276d1 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:44587: session id = c9f609481ded4d1aad9dd5f2341fd931-50b7150bbeb645d1b00d1a561c3276d1
I20240501 08:12:42.845609 10772 tablet_copy_source_session.cc:205] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.846516 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:42.866209 14666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b7150bbeb645d1b00d1a561c3276d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.870173 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:42.871928 10731 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:42.872324 10732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:42.878908 14666 tablet_copy_client.cc:783] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.879223 14666 tablet_copy_client.cc:647] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.886843 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.887679 14548 raft_consensus.cc:1050] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:42.888756 14548 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:42.897104 14666 tablet_copy_client.cc:523] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.897464 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.901432 14666 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:42.902678 14700 ts_tablet_manager.cc:902] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:42.914772 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:42.918406 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.935149 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.936647 12404 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:42.937213 14567 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:42.944041 10751 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:42.944399 10487 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:42.944751 14548 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:42.945026 13382 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:42.960103 12274 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:42.960881 13382 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:42.979460 14567 raft_consensus.cc:1013] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer b6a964e54f3b44eda6d0666e0c1e4bea: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:12:43.007104 12665 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:43.017343 13382 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.020107 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:43.024256 12404 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.025146 10751 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.025611 12272 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.045223 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:43.051920 10488 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.055460 14567 raft_consensus.cc:1050] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:43.056353 14567 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:43.062219 12403 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.063831 13382 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.067698 14700 tablet_copy_client.cc:308] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:43.068437 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 21065b7dcc18456eb1b4c2376d51eec4 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:51519)
I20240501 08:12:43.068596 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 21065b7dcc18456eb1b4c2376d51eec4 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:51519: session id = b6a964e54f3b44eda6d0666e0c1e4bea-21065b7dcc18456eb1b4c2376d51eec4
I20240501 08:12:43.070134 12272 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.070909 13382 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.071427 10772 tablet_copy_source_session.cc:205] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:43.073266 13382 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.074331 14700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21065b7dcc18456eb1b4c2376d51eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:43.079944 14364 raft_consensus.cc:1050] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:43.089514 14364 consensus_queue.cc:237] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:43.098081 10487 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.099268 14548 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.100536 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:43.101238 10487 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.105152 12404 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.106317 10221 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 5 to 6, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.120580 12272 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.121312 14359 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.129320 10487 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.130333 14166 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.144654 10361 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.148598 14166 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.151973 10751 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.152709 14720 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.154594 10751 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.163393 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:43.191141 12273 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.226498 14700 tablet_copy_client.cc:783] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:43.226900 14700 tablet_copy_client.cc:647] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:43.232187 14700 tablet_copy_client.cc:523] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:43.240588 14723 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.241556 12272 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.248397 14730 ts_tablet_manager.cc:902] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:43.254575 10487 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.260444 10361 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.281294 10217 catalog_manager.cc:5484] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 4 to 5, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.289211 14700 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:43.294319 10220 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:43.311836 14730 tablet_copy_client.cc:308] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:43.312533 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 4799a93d96a64a75b1984bb85fcad7e8 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:48051)
I20240501 08:12:43.312705 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 4799a93d96a64a75b1984bb85fcad7e8 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:48051: session id = bb2f0099fb794400b5c24927e213e5d8-4799a93d96a64a75b1984bb85fcad7e8
I20240501 08:12:43.315712 10512 tablet_copy_source_session.cc:205] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:43.317323 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4799a93d96a64a75b1984bb85fcad7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:43.326558 14730 tablet_copy_client.cc:783] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:43.326812 14730 tablet_copy_client.cc:647] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:43.327749 14631 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:43.338275 14666 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:43.338876 14666 tablet_bootstrap.cc:492] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:43.339242 14666 ts_tablet_manager.cc:1367] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.438s	user 0.048s	sys 0.010s
I20240501 08:12:43.344935 14666 raft_consensus.cc:348] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:43.349516 14666 raft_consensus.cc:729] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:43.349853 14666 consensus_queue.cc:260] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:43.350765 14666 ts_tablet_manager.cc:1398] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.000s
I20240501 08:12:43.352741 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-50b7150bbeb645d1b00d1a561c3276d1 received from {username='slave'} at 127.2.248.136:44587
I20240501 08:12:43.352890 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-50b7150bbeb645d1b00d1a561c3276d1 on tablet 50b7150bbeb645d1b00d1a561c3276d1 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:43.361135 10622 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:43.364990 10752 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:43.365981 14718 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:43.366452 12403 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:43.367156 14718 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:43.368698 10752 consensus_queue.cc:237] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:43.371138 10487 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.372618 14364 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.379734 14631 tablet_bootstrap.cc:492] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:43.380074 14631 ts_tablet_manager.cc:1367] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.781s	user 0.070s	sys 0.048s
I20240501 08:12:43.381114 14631 raft_consensus.cc:348] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:43.381389 14631 raft_consensus.cc:729] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:43.381659 14631 consensus_queue.cc:260] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:43.381695 10487 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.382251 14631 ts_tablet_manager.cc:1398] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:12:43.385051 10642 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-cf1f28d15c094fd98fac1b91d2785744 received from {username='slave'} at 127.2.248.136:56369
I20240501 08:12:43.385214 10642 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-cf1f28d15c094fd98fac1b91d2785744 on tablet cf1f28d15c094fd98fac1b91d2785744 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:43.379822 14723 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.394073 14718 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.396850 10752 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.407457 10219 catalog_manager.cc:5484] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.411651 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b924c6344714a92ac37ae28961b02f5 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.413000 12272 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.414033 12272 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:43.414484 12403 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.415140 14723 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.417428 12272 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.418555 14720 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:43.419045 14730 tablet_copy_client.cc:523] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:43.422468 12274 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.429023 10219 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 7 to 8, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:43.434048 14650 raft_consensus.cc:1050] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:43.434967 14650 consensus_queue.cc:237] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:43.439766 12665 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.440873 10751 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.440902 14486 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.445149 14492 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.445665 14531 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:43.446136 14531 tablet_bootstrap.cc:492] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Bootstrap complete.
I20240501 08:12:43.446484 14531 ts_tablet_manager.cc:1367] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Time spent bootstrapping tablet: real 1.213s	user 0.058s	sys 0.006s
W20240501 08:12:43.447381 10427 consensus_peers.cc:485] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.449071 10622 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.449685 14486 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.452755 14486 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:43.455155 10752 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:43.456655 10622 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:43.457182 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a63e16f11894c2886091dcd3d204106 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.460702 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a63e16f11894c2886091dcd3d204106 with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:43.470155 12665 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:43.476461 10622 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:43.479035 10751 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:43.479714 14720 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:43.483203 14720 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.485265 14730 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:43.487480 10751 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.491998 10217 catalog_manager.cc:5484] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 5 to 6, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.494938 12403 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:43.495409 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b924c6344714a92ac37ae28961b02f5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.495674 14720 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:43.498462 12403 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.503583 10215 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 8 to 9, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.503818 10487 consensus_queue.cc:237] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:43.508642 12665 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.509290 14650 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.510386 10622 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.511600 14650 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.512677 10218 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 4b924c6344714a92ac37ae28961b02f5 with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8
I20240501 08:12:43.513551 12254 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b924c6344714a92ac37ae28961b02f5 with delete_type TABLET_DATA_TOMBSTONED (TS bb2f0099fb794400b5c24927e213e5d8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53270
I20240501 08:12:43.514667 14486 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:43.521282 12666 raft_consensus.cc:1203] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 08:12:43.521281 10215 catalog_manager.cc:5484] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:43.521968 10619 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:43.529453 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4799a93d96a64a75b1984bb85fcad7e8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.530217 12666 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:43.530949 12253 tablet_service.cc:1514] Processing DeleteTablet for tablet 4799a93d96a64a75b1984bb85fcad7e8 with delete_type TABLET_DATA_TOMBSTONED (TS bb2f0099fb794400b5c24927e213e5d8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53270
I20240501 08:12:43.532581 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b924c6344714a92ac37ae28961b02f5 with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:43.553094 14754 tablet_replica.cc:330] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:43.556174 14754 raft_consensus.cc:2229] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.557576 14754 raft_consensus.cc:2258] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.560526 14531 raft_consensus.cc:348] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } }
I20240501 08:12:43.561463 14756 tablet_replica.cc:330] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:43.561666 14531 raft_consensus.cc:729] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Initialized, Role: LEARNER
I20240501 08:12:43.561933 14531 consensus_queue.cc:260] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: true } }
I20240501 08:12:43.564006 14756 raft_consensus.cc:2229] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.564523 14756 raft_consensus.cc:2258] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.564751 14531 ts_tablet_manager.cc:1398] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed: Time spent starting tablet: real 0.118s	user 0.004s	sys 0.000s
I20240501 08:12:43.565419 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session c3747ec2a93c4684962f215616e896ed-af24b8ab4d0c43ef986d89d4aefc263c received from {username='slave'} at 127.2.248.129:40069
I20240501 08:12:43.565608 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session c3747ec2a93c4684962f215616e896ed-af24b8ab4d0c43ef986d89d4aefc263c on tablet af24b8ab4d0c43ef986d89d4aefc263c with peer c3747ec2a93c4684962f215616e896ed
I20240501 08:12:43.566318 12666 raft_consensus.cc:1203] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240501 08:12:43.569550 14756 ts_tablet_manager.cc:1875] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.578775 14578 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:43.579226 14578 tablet_bootstrap.cc:492] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:43.579526 14578 ts_tablet_manager.cc:1367] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 0.965s	user 0.054s	sys 0.004s
I20240501 08:12:43.581061 14755 tablet_replica.cc:330] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: stopping tablet replica
I20240501 08:12:43.585399 14755 raft_consensus.cc:2229] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.585682 14754 ts_tablet_manager.cc:1875] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.587906 14756 ts_tablet_manager.cc:1888] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:43.588506 14756 log.cc:1196] T 4b924c6344714a92ac37ae28961b02f5 P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b924c6344714a92ac37ae28961b02f5
I20240501 08:12:43.589413 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet 4b924c6344714a92ac37ae28961b02f5 (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:43.591900 10751 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.593628 14753 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.596026 14578 raft_consensus.cc:348] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:43.596323 14578 raft_consensus.cc:729] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:43.596028 10751 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:43.596532 14578 consensus_queue.cc:260] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:43.600304 14700 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:43.600864 14700 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:43.601260 14700 ts_tablet_manager.cc:1367] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.312s	user 0.041s	sys 0.009s
I20240501 08:12:43.604173 14700 raft_consensus.cc:348] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:43.604302 14755 raft_consensus.cc:2258] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.604426 14700 raft_consensus.cc:729] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:43.604658 14700 consensus_queue.cc:260] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:43.605396 14700 ts_tablet_manager.cc:1398] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:12:43.606420 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-21065b7dcc18456eb1b4c2376d51eec4 received from {username='slave'} at 127.2.248.134:51519
I20240501 08:12:43.606592 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-21065b7dcc18456eb1b4c2376d51eec4 on tablet 21065b7dcc18456eb1b4c2376d51eec4 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:43.608546 14578 ts_tablet_manager.cc:1398] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.000s
I20240501 08:12:43.609544 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-3453d8d883014219b1b14c88e250736b received from {username='slave'} at 127.2.248.135:48525
I20240501 08:12:43.609705 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-3453d8d883014219b1b14c88e250736b on tablet 3453d8d883014219b1b14c88e250736b with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:43.614885 14755 ts_tablet_manager.cc:1875] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.622182 14754 ts_tablet_manager.cc:1888] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:43.622324 14754 log.cc:1196] T 2a63e16f11894c2886091dcd3d204106 P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a63e16f11894c2886091dcd3d204106
I20240501 08:12:43.623200 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet 2a63e16f11894c2886091dcd3d204106 (table rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]) successfully deleted
I20240501 08:12:43.625852 14755 ts_tablet_manager.cc:1888] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:12:43.626454 14755 log.cc:1196] T 4b924c6344714a92ac37ae28961b02f5 P bb2f0099fb794400b5c24927e213e5d8: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/wals/4b924c6344714a92ac37ae28961b02f5
I20240501 08:12:43.628906 10207 catalog_manager.cc:4830] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): tablet 4b924c6344714a92ac37ae28961b02f5 (table rebalance_test_table_1 [id=2a2bb2d036ee4460892b76fd8cda6ceb]) successfully deleted
I20240501 08:12:43.631054 10487 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:12:43.632488 10207 catalog_manager.cc:4810] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): delete failed for tablet 4799a93d96a64a75b1984bb85fcad7e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4799a93d96a64a75b1984bb85fcad7e8 already in progress: copying tablet
W20240501 08:12:43.634786 10425 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 185a62881ea84131a0c14987e4296338. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.635483 12534 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.635613 10751 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.636204 14486 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.637738 14564 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:43.637892 14486 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.638170 14564 tablet_bootstrap.cc:492] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:43.638439 14564 ts_tablet_manager.cc:1367] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.986s	user 0.060s	sys 0.000s
I20240501 08:12:43.639119 14564 raft_consensus.cc:348] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:43.639405 14564 raft_consensus.cc:729] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:43.639627 14564 consensus_queue.cc:260] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:43.651129 14564 ts_tablet_manager.cc:1398] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I20240501 08:12:43.652298 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-f5a58b7166534844b0963f4fab46dfd9 received from {username='slave'} at 127.2.248.134:46779
I20240501 08:12:43.652470 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-f5a58b7166534844b0963f4fab46dfd9 on tablet f5a58b7166534844b0963f4fab46dfd9 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:43.655238 14486 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:43.661473 10752 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:43.662149 10212 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 6 to 7, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.680214 12536 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:12:43.746637 10622 consensus_queue.cc:237] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
W20240501 08:12:43.761219 10557 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbef0147276d410da3d3830e682efe6d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.761363 12536 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.762027 14567 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.764358 10361 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.766789 14567 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.787397 10752 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:43.790851 12403 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.791430 14723 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.791826 10622 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:12:43.791889 10688 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21065b7dcc18456eb1b4c2376d51eec4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.792330 14723 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.794652 14723 raft_consensus.cc:1013] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: attempt to promote peer b6a964e54f3b44eda6d0666e0c1e4bea: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:12:43.799774 14723 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.807550 12403 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.809141 12534 raft_consensus.cc:1203] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:43.810734 10622 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.816087 14166 raft_consensus.cc:1050] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:43.816970 14166 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:43.824637 10688 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21065b7dcc18456eb1b4c2376d51eec4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.825057 12403 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.825604 14166 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.815521 10219 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.827318 10619 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.827986 14166 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.830977 14359 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.837428 12403 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.846164 10619 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.849998 10215 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.854497 10752 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:43.857092 12404 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.857785 14723 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.861135 10622 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.861760 14723 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.864681 14364 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.865355 10619 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.868676 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21065b7dcc18456eb1b4c2376d51eec4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.871351 12404 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.878211 10219 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.886915 10358 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.888362 10358 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.889021 10358 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.891986 14723 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.897145 10358 raft_consensus.cc:858] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:43.898797 10358 raft_consensus.cc:1565] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:43.901501 10621 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:43.919487 10359 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:43.928396 10362 raft_consensus.cc:1203] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
I20240501 08:12:43.937886 14718 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:43.939394 10361 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:43.947762 10220 catalog_manager.cc:5484] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:43.956246 12536 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:43.958693 10619 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.959604 10487 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.959997 14375 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.960165 12516 tablet_service.cc:1514] Processing DeleteTablet for tablet 21065b7dcc18456eb1b4c2376d51eec4 with delete_type TABLET_DATA_TOMBSTONED (TS 441e281fafbf42f8938c3f2f42ff7527 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:32994
I20240501 08:12:43.960350 12873 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:43.961269 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03d0f89e1d8f40fe84d275b117d8145f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.961292 12666 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.935225 10358 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
W20240501 08:12:43.963574 10298 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d4ab8e42a484cf89b42f5f53f511265. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.964180 10487 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:43.964800 14720 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.965441 14720 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:43.970465 10358 raft_consensus.cc:858] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
W20240501 08:12:43.970619 10358 raft_consensus.cc:1565] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:43.975453 10358 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:43.982839 10752 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:43.985417 12403 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:43.986181 14364 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:43.988549 10619 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:43.989377 14364 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:43.990068 14364 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.993326 12403 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:43.999235 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21065b7dcc18456eb1b4c2376d51eec4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 08:12:44.000885 10208 catalog_manager.cc:4797] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): delete failed for tablet 21065b7dcc18456eb1b4c2376d51eec4 because the tablet was not found. No further retry: Not found: Tablet not found: 21065b7dcc18456eb1b4c2376d51eec4
I20240501 08:12:44.001981 14720 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:44.011188 10218 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.013947 12873 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.016607 10619 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3747ec2a93c4684962f215616e896ed (127.2.248.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.017578 14564 ts_tablet_manager.cc:902] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:44.022490 10341 tablet_service.cc:1514] Processing DeleteTablet for tablet 21065b7dcc18456eb1b4c2376d51eec4 with delete_type TABLET_DATA_TOMBSTONED (TS c3747ec2a93c4684962f215616e896ed not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33172
I20240501 08:12:44.024389 14754 tablet_replica.cc:330] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: stopping tablet replica
I20240501 08:12:44.024610 14754 raft_consensus.cc:2229] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:44.024982 14754 raft_consensus.cc:2258] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:44.026890 10218 catalog_manager.cc:5484] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.027606 10219 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.031342 10487 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.035331 12666 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:44.040491 10492 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:44.043771 14754 ts_tablet_manager.cc:1875] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.053256 14754 ts_tablet_manager.cc:1888] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:12:44.053550 14754 log.cc:1196] T 21065b7dcc18456eb1b4c2376d51eec4 P c3747ec2a93c4684962f215616e896ed: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/21065b7dcc18456eb1b4c2376d51eec4
I20240501 08:12:44.054510 10207 catalog_manager.cc:4830] TS c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): tablet 21065b7dcc18456eb1b4c2376d51eec4 (table rebalance_test_table_9 [id=3249ecbd03094e94a93017efa42b105c]) successfully deleted
I20240501 08:12:44.059918 12403 raft_consensus.cc:1203] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:44.064014 10621 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.086792 14718 raft_consensus.cc:1050] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:44.087911 14718 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:44.094348 12666 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:44.095048 14720 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:44.097919 12403 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:44.098055 10752 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:44.098507 14567 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:44.100535 14720 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:44.114852 14564 tablet_copy_client.cc:308] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:44.116391 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 185a62881ea84131a0c14987e4296338 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:45463)
I20240501 08:12:44.116575 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 185a62881ea84131a0c14987e4296338 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:45463: session id = b6a964e54f3b44eda6d0666e0c1e4bea-185a62881ea84131a0c14987e4296338
I20240501 08:12:44.119494 10512 tablet_copy_source_session.cc:205] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.121582 14564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185a62881ea84131a0c14987e4296338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.124578 10621 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:44.128059 14564 tablet_copy_client.cc:783] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.130060 14564 tablet_copy_client.cc:647] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.130416 14567 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.136672 12536 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:44.137814 14567 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:44.138880 10215 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.139395 12403 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.144560 14564 tablet_copy_client.cc:523] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.148065 14564 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:12:44.148495 10752 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.156239 10619 consensus_queue.cc:237] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:44.158869 12403 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:44.159780 14720 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:44.164294 14718 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.164801 12403 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.168362 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf1f28d15c094fd98fac1b91d2785744 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:44.171459 10218 catalog_manager.cc:5484] T cf1f28d15c094fd98fac1b91d2785744 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 10 to 11, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.179112 10732 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1f28d15c094fd98fac1b91d2785744 with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:44.161854 12666 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.178367 10492 consensus_queue.cc:237] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:44.183274 12665 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:44.183624 10751 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:44.183900 14753 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:44.184329 14720 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:12:44.185084 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6624441a1f1143648913595f905f8039. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.186543 14753 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.188946 12665 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.195858 14720 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:44.203572 10218 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.204099 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4799a93d96a64a75b1984bb85fcad7e8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:44.207422 10752 raft_consensus.cc:1261] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:44.210551 10215 catalog_manager.cc:5484] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:44.212360 14753 consensus_queue.cc:1035] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:44.215113 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet 4799a93d96a64a75b1984bb85fcad7e8 with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:44.222726 10752 raft_consensus.cc:2932] T 4799a93d96a64a75b1984bb85fcad7e8 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.232568 14723 raft_consensus.cc:1050] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:44.233559 14723 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:44.200667 12536 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:44.241595 10750 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:44.248808 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.251358 12667 raft_consensus.cc:1261] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:44.252408 14720 consensus_queue.cc:1035] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:44.255995 12667 raft_consensus.cc:2932] T cf1f28d15c094fd98fac1b91d2785744 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.266971 14650 raft_consensus.cc:1050] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:44.267786 14650 consensus_queue.cc:237] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:44.272181 12272 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:44.272848 14364 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:44.273036 14876 tablet_replica.cc:330] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:44.273250 14876 raft_consensus.cc:2229] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:44.274261 14876 raft_consensus.cc:2258] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:44.274875 12403 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:44.275415 14876 ts_tablet_manager.cc:1875] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.275624 10358 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:44.275857 10621 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:44.276202 14364 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:44.281955 12534 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.282280 10359 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.282920 14650 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.283279 14753 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.293627 14874 ts_tablet_manager.cc:902] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:44.299674 10621 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.300356 14650 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.304507 14866 tablet_replica.cc:330] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:44.304950 14866 raft_consensus.cc:2229] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:44.305353 14866 raft_consensus.cc:2258] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:44.307639 14364 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:44.308230 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:44.308601 14364 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:44.310194 10492 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:44.310278 12666 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:44.310914 14364 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:44.314213 14866 ts_tablet_manager.cc:1875] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.320905 14723 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:44.335256 10732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:44.344554 10751 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:44.353924 14364 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.355648 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.356809 12274 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.357280 14874 tablet_copy_client.cc:308] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:44.360037 12667 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.364650 14753 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.365388 10219 catalog_manager.cc:5484] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:44.369197 10622 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.370502 10749 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:44.374485 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet bbef0147276d410da3d3830e682efe6d from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56369)
I20240501 08:12:44.374660 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet bbef0147276d410da3d3830e682efe6d from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56369: session id = c9f609481ded4d1aad9dd5f2341fd931-bbef0147276d410da3d3830e682efe6d
I20240501 08:12:44.377456 10642 tablet_copy_source_session.cc:205] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.378131 10359 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.380025 14874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbef0147276d410da3d3830e682efe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.378753 10218 catalog_manager.cc:5484] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.383165 12667 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:44.383754 14364 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:44.387387 14166 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.391651 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50b7150bbeb645d1b00d1a561c3276d1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:44.393910 12666 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.395134 10217 catalog_manager.cc:5484] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:44.403710 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b7150bbeb645d1b00d1a561c3276d1 with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:44.409875 14364 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.410195 10361 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.415362 12272 raft_consensus.cc:1261] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:44.416042 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:44.416723 14166 consensus_queue.cc:1035] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:44.419173 12274 raft_consensus.cc:2932] T 50b7150bbeb645d1b00d1a561c3276d1 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:44.422456 12403 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.423666 10220 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:44.425256 10487 consensus_queue.cc:237] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:44.428678 10361 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:44.429217 10749 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:44.429380 14486 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:12:44.432258 10688 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd01dd912e1e4ca9a104270f3cd004b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.432056 14753 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.433522 10361 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:44.434087 14723 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:44.437463 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3453d8d883014219b1b14c88e250736b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:44.439133 10220 catalog_manager.cc:5484] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.442031 10619 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:44.443151 12534 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.443722 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet 3453d8d883014219b1b14c88e250736b with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:44.444356 10361 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.445396 12404 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:44.449414 14359 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:44.450412 14359 raft_consensus.cc:1050] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER c3747ec2a93c4684962f215616e896ed to VOTER
I20240501 08:12:44.451359 14359 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } }
I20240501 08:12:44.454598 14906 tablet_replica.cc:330] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:44.454813 14906 raft_consensus.cc:2229] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:44.455137 14906 pending_rounds.cc:62] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Trying to abort 1 pending ops.
I20240501 08:12:44.455214 14906 pending_rounds.cc:68] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022801568698085376 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "50b7150bbeb645d1b00d1a561c3276d1" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } } }
I20240501 08:12:44.455631 14906 raft_consensus.cc:2894] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 08:12:44.455516 14876 ts_tablet_manager.cc:1888] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:44.455751 14906 raft_consensus.cc:2866] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:12:44.455868 14906 raft_consensus.cc:2258] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:44.455909 14876 log.cc:1196] T 4799a93d96a64a75b1984bb85fcad7e8 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/4799a93d96a64a75b1984bb85fcad7e8
I20240501 08:12:44.456777 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet 4799a93d96a64a75b1984bb85fcad7e8 (table rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]) successfully deleted
I20240501 08:12:44.457252 14876 tablet_replica.cc:330] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:44.457499 14876 raft_consensus.cc:2229] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:44.457947 14876 raft_consensus.cc:2258] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 08:12:44.464954 14898 consensus_queue.cc:1159] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "861f629e2d134ff08f25fa553dfb5bda" status { last_received { term: 1 index: 8 } last_committed_idx: 7 last_received_current_leader { term: 1 index: 8 } }
I20240501 08:12:44.465899 14866 ts_tablet_manager.cc:1888] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:12:44.466116 14866 log.cc:1196] T cf1f28d15c094fd98fac1b91d2785744 P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf1f28d15c094fd98fac1b91d2785744
I20240501 08:12:44.466572 14906 ts_tablet_manager.cc:1875] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.466879 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet cf1f28d15c094fd98fac1b91d2785744 (table rebalance_test_table_8 [id=93d41bd6b8824bf68f8a23fecdbd5d89]) successfully deleted
I20240501 08:12:44.469844 14723 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:44.471556 10619 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.481220 12536 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:44.481842 14753 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:44.483584 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.483388 12535 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:44.493973 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:44.494180 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.495726 14723 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.502339 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cd01dd912e1e4ca9a104270f3cd004b1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:12:44.503363 10361 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.504316 10619 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.510471 10215 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.514446 14913 ts_tablet_manager.cc:902] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:44.518050 14906 ts_tablet_manager.cc:1888] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:12:44.521268 14730 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.521869 14730 tablet_bootstrap.cc:492] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:44.522778 14730 ts_tablet_manager.cc:1367] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 1.038s	user 0.056s	sys 0.011s
I20240501 08:12:44.523051 14906 log.cc:1196] T 50b7150bbeb645d1b00d1a561c3276d1 P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/50b7150bbeb645d1b00d1a561c3276d1
I20240501 08:12:44.527019 14876 ts_tablet_manager.cc:1875] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.528093 10361 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:12:44.528755 14364 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:12:44.529278 12404 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:12:44.529893 14730 raft_consensus.cc:348] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:44.530210 14730 raft_consensus.cc:729] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:44.530444 14730 consensus_queue.cc:260] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:44.531885 14723 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:12:44.533457 14730 ts_tablet_manager.cc:1398] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240501 08:12:44.536226 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet 50b7150bbeb645d1b00d1a561c3276d1 (table rebalance_test_table_0 [id=62b3aca374c44c509e954361e2dcbb20]) successfully deleted
I20240501 08:12:44.525974 12403 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.548951 12254 tablet_service.cc:1514] Processing DeleteTablet for tablet 4799a93d96a64a75b1984bb85fcad7e8 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:53270
I20240501 08:12:44.550149 12274 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:12:44.551610 14364 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:12:44.553887 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-4799a93d96a64a75b1984bb85fcad7e8 received from {username='slave'} at 127.2.248.133:48051
I20240501 08:12:44.554056 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-4799a93d96a64a75b1984bb85fcad7e8 on tablet 4799a93d96a64a75b1984bb85fcad7e8 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:44.573824 14921 tablet_replica.cc:330] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: stopping tablet replica
I20240501 08:12:44.575125 14921 raft_consensus.cc:2229] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:44.575428 14921 raft_consensus.cc:2258] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:44.576395 14921 ts_tablet_manager.cc:1875] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.576844 14919 ts_tablet_manager.cc:902] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:44.585559 10751 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:44.587090 14723 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c3747ec2a93c4684962f215616e896ed (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
I20240501 08:12:44.587729 14874 tablet_copy_client.cc:783] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.588029 14874 tablet_copy_client.cc:647] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.587988 12274 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c3747ec2a93c4684962f215616e896ed (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
W20240501 08:12:44.589957 10688 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd01dd912e1e4ca9a104270f3cd004b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.590211 14874 tablet_copy_client.cc:523] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.594727 10358 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c3747ec2a93c4684962f215616e896ed (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
I20240501 08:12:44.596799 10217 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 13 to 14, c3747ec2a93c4684962f215616e896ed (127.2.248.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
I20240501 08:12:44.604312 10361 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:44.604959 14359 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:44.607409 14166 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.611585 10359 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:44.611878 10361 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.617653 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd01dd912e1e4ca9a104270f3cd004b1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:44.619179 10217 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.624563 14375 raft_consensus.cc:1050] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
I20240501 08:12:44.625505 14375 consensus_queue.cc:237] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:44.626251 10600 tablet_service.cc:1514] Processing DeleteTablet for tablet cd01dd912e1e4ca9a104270f3cd004b1 with delete_type TABLET_DATA_TOMBSTONED (TS 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:40014
I20240501 08:12:44.633338 12401 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 08:12:44.634018 10297 consensus_peers.cc:485] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d188d415f63f448195ae90e8e3d66e0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.634294 14830 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:44.662223 14913 tablet_copy_client.cc:308] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:44.663795 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 6d4ab8e42a484cf89b42f5f53f511265 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:55933)
I20240501 08:12:44.663990 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 6d4ab8e42a484cf89b42f5f53f511265 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:55933: session id = 441e281fafbf42f8938c3f2f42ff7527-6d4ab8e42a484cf89b42f5f53f511265
I20240501 08:12:44.666925 10381 tablet_copy_source_session.cc:205] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.668829 14913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4ab8e42a484cf89b42f5f53f511265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.678571 14913 tablet_copy_client.cc:783] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.678921 14913 tablet_copy_client.cc:647] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.682260 14913 tablet_copy_client.cc:523] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.688720 12402 raft_consensus.cc:1261] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.689561 10619 raft_consensus.cc:1261] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.690171 12873 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.690691 12873 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.692072 14730 ts_tablet_manager.cc:902] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:44.693011 14913 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
W20240501 08:12:44.694669 14920 log.cc:924] Time spent T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20240501 08:12:44.698256 12404 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, c3747ec2a93c4684962f215616e896ed (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
I20240501 08:12:44.722061 10732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:12:44.731998 14932 log.cc:924] Time spent T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240501 08:12:44.733235 12873 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:44.736908 12401 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:44.748822 10220 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.751156 12405 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:44.754758 14723 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:44.769740 10621 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
W20240501 08:12:44.773844 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:44.774227 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6624441a1f1143648913595f905f8039. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.775305 10749 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:44.775524 10492 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:44.775979 14720 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:44.777266 14830 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:44.780493 12536 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:44.781651 14720 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:44.784453 14863 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:44.785972 10492 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:44.785643 10749 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:44.789288 12536 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:44.800570 10220 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 10 to 11, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:44.769876 12403 raft_consensus.cc:2932] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:44.813596 14830 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.819643 12402 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.829483 14874 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:44.850368 14876 ts_tablet_manager.cc:1888] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:44.850626 14876 log.cc:1196] T 3453d8d883014219b1b14c88e250736b P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/3453d8d883014219b1b14c88e250736b
I20240501 08:12:44.851644 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet 3453d8d883014219b1b14c88e250736b (table rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]) successfully deleted
I20240501 08:12:44.852486 14876 tablet_replica.cc:330] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: stopping tablet replica
I20240501 08:12:44.852816 14876 raft_consensus.cc:2229] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:44.854873 14730 tablet_copy_client.cc:308] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:44.877099 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.877280 10220 catalog_manager.cc:5484] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.892486 14919 tablet_copy_client.cc:308] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:44.894212 10751 raft_consensus.cc:1261] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.895362 14830 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.910573 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:44.919584 10749 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:44.855548 10642 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 6624441a1f1143648913595f905f8039 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47709)
I20240501 08:12:44.898126 10619 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.931752 10642 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 6624441a1f1143648913595f905f8039 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47709: session id = bb2f0099fb794400b5c24927e213e5d8-6624441a1f1143648913595f905f8039
I20240501 08:12:44.935209 10642 tablet_copy_source_session.cc:205] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.899333 10641 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 03d0f89e1d8f40fe84d275b117d8145f from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47709)
I20240501 08:12:44.935672 10641 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 03d0f89e1d8f40fe84d275b117d8145f from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47709: session id = bb2f0099fb794400b5c24927e213e5d8-03d0f89e1d8f40fe84d275b117d8145f
I20240501 08:12:44.938361 10641 tablet_copy_source_session.cc:205] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.938580 14876 raft_consensus.cc:2258] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:44.939631 14876 ts_tablet_manager.cc:1875] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.942047 12274 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:44.942615 14723 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:12:44.943275 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a6ed4ea6d6941d2af0f81eaf744bcea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.943837 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.944202 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d0f89e1d8f40fe84d275b117d8145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.945549 10358 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:44.946168 14723 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:44.955463 14919 tablet_copy_client.cc:783] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.955727 14919 tablet_copy_client.cc:647] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.958714 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.959232 14919 tablet_copy_client.cc:523] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.961182 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:44.967561 14919 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:44.968375 14730 tablet_copy_client.cc:783] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.968680 14730 tablet_copy_client.cc:647] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.970618 14730 tablet_copy_client.cc:523] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.974241 14730 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:44.976897 10751 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:44.983366 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:44.992857 10361 consensus_queue.cc:237] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:44.996709 12402 raft_consensus.cc:1261] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:44.997295 14830 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:44.997495 10619 raft_consensus.cc:1261] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:44.999243 14830 consensus_queue.cc:1035] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:45.005283 14965 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:45.006111 10619 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:45.015075 12402 raft_consensus.cc:2932] T f5a58b7166534844b0963f4fab46dfd9 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:45.018901 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f5a58b7166534844b0963f4fab46dfd9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:45.029253 10215 catalog_manager.cc:5484] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 5 to 6, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:45.033850 10732 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a58b7166534844b0963f4fab46dfd9 with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:45.044404 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:45.057621 14980 tablet_replica.cc:330] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:45.058025 14980 raft_consensus.cc:2229] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.058622 14980 raft_consensus.cc:2258] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.065873 14980 ts_tablet_manager.cc:1875] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:45.070091 14921 ts_tablet_manager.cc:1888] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:45.070384 14921 log.cc:1196] T 4799a93d96a64a75b1984bb85fcad7e8 P bb2f0099fb794400b5c24927e213e5d8: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/wals/4799a93d96a64a75b1984bb85fcad7e8
I20240501 08:12:45.073293 10207 catalog_manager.cc:4830] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): tablet 4799a93d96a64a75b1984bb85fcad7e8 (table rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]) successfully deleted
I20240501 08:12:45.078410 14979 ts_tablet_manager.cc:902] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:45.088443 14364 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:45.093439 12272 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:45.102762 10215 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:45.118731 14980 ts_tablet_manager.cc:1888] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:45.118916 14980 log.cc:1196] T f5a58b7166534844b0963f4fab46dfd9 P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/f5a58b7166534844b0963f4fab46dfd9
I20240501 08:12:45.120415 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet f5a58b7166534844b0963f4fab46dfd9 (table rebalance_test_table_3 [id=6b8a9eb2e539416a8cfa53d67bb90b65]) successfully deleted
I20240501 08:12:45.132716 10358 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:45.154475 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:45.194696 14979 tablet_copy_client.cc:308] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:45.195549 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet cd01dd912e1e4ca9a104270f3cd004b1 from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:43581)
I20240501 08:12:45.195756 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet cd01dd912e1e4ca9a104270f3cd004b1 from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:43581: session id = 441e281fafbf42f8938c3f2f42ff7527-cd01dd912e1e4ca9a104270f3cd004b1
I20240501 08:12:45.202149 10772 tablet_copy_source_session.cc:205] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.203552 14979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd01dd912e1e4ca9a104270f3cd004b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.237062 10619 consensus_queue.cc:237] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:45.246421 14996 ts_tablet_manager.cc:902] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:12:45.273725 14876 ts_tablet_manager.cc:1888] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:12:45.283473 14979 tablet_copy_client.cc:783] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.283879 14979 tablet_copy_client.cc:647] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.283885 10492 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:45.284436 10358 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:45.304491 14564 log.cc:924] Time spent T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: Append to log took a long time: real 0.084s	user 0.004s	sys 0.000s
I20240501 08:12:45.285712 14863 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:45.309250 10558 consensus_peers.cc:485] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84789f1228f64be5b6976a9d52f64bd0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:45.310235 14862 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:45.311940 14996 tablet_copy_client.cc:308] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:12:45.312718 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet d188d415f63f448195ae90e8e3d66e0e from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56961)
I20240501 08:12:45.312868 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet d188d415f63f448195ae90e8e3d66e0e from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56961: session id = c9f609481ded4d1aad9dd5f2341fd931-d188d415f63f448195ae90e8e3d66e0e
I20240501 08:12:45.315800 10381 tablet_copy_source_session.cc:205] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.320981 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d188d415f63f448195ae90e8e3d66e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.339282 14996 tablet_copy_client.cc:783] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.339941 14996 tablet_copy_client.cc:647] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.340144 14876 log.cc:1196] T cd01dd912e1e4ca9a104270f3cd004b1 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/cd01dd912e1e4ca9a104270f3cd004b1
I20240501 08:12:45.341288 10205 catalog_manager.cc:4830] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): tablet cd01dd912e1e4ca9a104270f3cd004b1 (table rebalance_test_table_5 [id=169807e07dfe479c83d8666ceedc07a5]) successfully deleted
I20240501 08:12:45.344359 14979 tablet_copy_client.cc:523] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.346681 14996 tablet_copy_client.cc:523] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.348919 15007 ts_tablet_manager.cc:902] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:45.349773 10487 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:45.347738 14862 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:45.361793 14996 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:45.366793 10220 catalog_manager.cc:5484] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:45.375320 14730 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.375846 14730 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:45.376133 14730 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.402s	user 0.053s	sys 0.011s
I20240501 08:12:45.376969 14730 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.377255 14730 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:45.377547 14730 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.378147 14730 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:12:45.379066 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-6624441a1f1143648913595f905f8039 received from {username='slave'} at 127.2.248.133:47709
I20240501 08:12:45.379228 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-6624441a1f1143648913595f905f8039 on tablet 6624441a1f1143648913595f905f8039 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:45.402666 14913 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.403100 14913 tablet_bootstrap.cc:492] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:45.403378 14913 ts_tablet_manager.cc:1367] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 0.710s	user 0.100s	sys 0.114s
I20240501 08:12:45.404059 14913 raft_consensus.cc:348] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:45.404346 14913 raft_consensus.cc:729] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:45.408244 14913 consensus_queue.cc:260] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:45.409154 14730 tablet_replica.cc:330] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: stopping tablet replica
I20240501 08:12:45.409417 14730 raft_consensus.cc:2229] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 08:12:45.409633 14730 raft_consensus.cc:2258] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 08:12:45.416260 14730 ts_tablet_manager.cc:1875] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:12:45.439016 15005 log.cc:924] Time spent T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20240501 08:12:45.448740 10359 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:45.460683 14730 ts_tablet_manager.cc:1888] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 08:12:45.460855 14730 log.cc:1196] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/wals/6624441a1f1143648913595f905f8039
I20240501 08:12:45.461438 14730 ts_tablet_manager.cc:902] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:12:45.471794 15013 ts_tablet_manager.cc:902] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:45.511015 14919 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.511647 14919 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:45.511979 14919 ts_tablet_manager.cc:1367] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.544s	user 0.066s	sys 0.000s
I20240501 08:12:45.512658 14979 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:45.518776 14919 raft_consensus.cc:348] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:45.519513 14919 raft_consensus.cc:729] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:45.519774 14919 consensus_queue.cc:260] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:45.521574 14919 ts_tablet_manager.cc:1398] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20240501 08:12:45.522500 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-03d0f89e1d8f40fe84d275b117d8145f received from {username='slave'} at 127.2.248.133:47709
I20240501 08:12:45.522646 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-03d0f89e1d8f40fe84d275b117d8145f on tablet 03d0f89e1d8f40fe84d275b117d8145f with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:45.525051 15007 tablet_copy_client.cc:308] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:45.525789 10641 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 6624441a1f1143648913595f905f8039 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56369)
I20240501 08:12:45.525979 10641 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 6624441a1f1143648913595f905f8039 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56369: session id = c9f609481ded4d1aad9dd5f2341fd931-6624441a1f1143648913595f905f8039
I20240501 08:12:45.529047 10641 tablet_copy_source_session.cc:205] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.530328 15007 ts_tablet_manager.cc:1875] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:12:45.531636 14913 ts_tablet_manager.cc:1398] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.128s	user 0.014s	sys 0.040s
I20240501 08:12:45.532449 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-6d4ab8e42a484cf89b42f5f53f511265 received from {username='slave'} at 127.2.248.135:55933
I20240501 08:12:45.532583 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-6d4ab8e42a484cf89b42f5f53f511265 on tablet 6d4ab8e42a484cf89b42f5f53f511265 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:45.542287 15007 ts_tablet_manager.cc:1888] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 08:12:45.542743 15007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.556972 15007 tablet_copy_client.cc:783] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.558328 15007 tablet_copy_client.cc:647] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.561125 15007 tablet_copy_client.cc:523] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.565346 12536 raft_consensus.cc:1203] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:12:45.573769 15007 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:12:45.583628 14874 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.590137 14874 tablet_bootstrap.cc:492] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:45.590551 14874 ts_tablet_manager.cc:1367] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.761s	user 0.064s	sys 0.009s
I20240501 08:12:45.595477 14874 raft_consensus.cc:348] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.596938 14874 raft_consensus.cc:729] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:45.597235 14874 consensus_queue.cc:260] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.618502 14874 ts_tablet_manager.cc:1398] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I20240501 08:12:45.621353 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-bbef0147276d410da3d3830e682efe6d received from {username='slave'} at 127.2.248.136:56369
I20240501 08:12:45.621503 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-bbef0147276d410da3d3830e682efe6d on tablet bbef0147276d410da3d3830e682efe6d with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:45.628947 15013 tablet_copy_client.cc:308] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:12:45.633011 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea from peer 441e281fafbf42f8938c3f2f42ff7527 ({username='slave'} at 127.2.248.135:43581)
I20240501 08:12:45.633246 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 8a6ed4ea6d6941d2af0f81eaf744bcea from peer 441e281fafbf42f8938c3f2f42ff7527 at {username='slave'} at 127.2.248.135:43581: session id = 441e281fafbf42f8938c3f2f42ff7527-8a6ed4ea6d6941d2af0f81eaf744bcea
I20240501 08:12:45.639945 10772 tablet_copy_source_session.cc:205] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.651227 15013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.657212 14996 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.659704 14996 tablet_bootstrap.cc:492] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:45.660099 14996 ts_tablet_manager.cc:1367] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.298s	user 0.057s	sys 0.004s
I20240501 08:12:45.661022 14996 raft_consensus.cc:348] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.661528 14996 raft_consensus.cc:729] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:45.661761 14996 consensus_queue.cc:260] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.666811 14996 ts_tablet_manager.cc:1398] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240501 08:12:45.667549 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-d188d415f63f448195ae90e8e3d66e0e received from {username='slave'} at 127.2.248.136:56961
I20240501 08:12:45.667721 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-d188d415f63f448195ae90e8e3d66e0e on tablet d188d415f63f448195ae90e8e3d66e0e with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:45.701496 15007 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.701933 15007 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:45.702206 15007 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.130s	user 0.054s	sys 0.002s
I20240501 08:12:45.703056 15007 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.703372 15007 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:45.703770 15007 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:45.706329 15007 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 08:12:45.707063 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-6624441a1f1143648913595f905f8039 received from {username='slave'} at 127.2.248.136:56369
I20240501 08:12:45.707204 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-6624441a1f1143648913595f905f8039 on tablet 6624441a1f1143648913595f905f8039 with peer c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:45.719242 15013 tablet_copy_client.cc:783] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.722844 10492 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:45.728663 14730 tablet_copy_client.cc:308] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:12:45.729725 10641 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 6624441a1f1143648913595f905f8039 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:47709)
I20240501 08:12:45.729951 10641 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 6624441a1f1143648913595f905f8039 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:47709: session id = bb2f0099fb794400b5c24927e213e5d8-6624441a1f1143648913595f905f8039
I20240501 08:12:45.734370 10641 tablet_copy_source_session.cc:205] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 08:12:45.735661 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfbef7af4781448bb04b107b53d58ca0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:45.734838 12667 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:45.736773 14486 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:45.739495 14730 ts_tablet_manager.cc:1875] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:12:45.741725 15013 tablet_copy_client.cc:647] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.746485 15013 tablet_copy_client.cc:523] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 441e281fafbf42f8938c3f2f42ff7527: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.769512 14730 ts_tablet_manager.cc:1888] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240501 08:12:45.772260 10619 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:45.773061 14486 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:45.786983 15013 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap starting.
I20240501 08:12:45.793587 14486 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:45.799434 10619 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:45.809012 10220 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:45.827711 14564 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.828321 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6624441a1f1143648913595f905f8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.836035 12667 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:45.850348 14564 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:12:45.851289 14564 ts_tablet_manager.cc:1367] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 1.703s	user 0.064s	sys 0.024s
I20240501 08:12:45.854694 14564 raft_consensus.cc:348] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:45.859055 14564 raft_consensus.cc:729] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:12:45.861302 14730 tablet_copy_client.cc:783] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.861625 14564 consensus_queue.cc:260] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:45.864464 14730 tablet_copy_client.cc:647] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.873941 14730 tablet_copy_client.cc:523] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.877897 14564 ts_tablet_manager.cc:1398] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.024s	user 0.002s	sys 0.000s
I20240501 08:12:45.880339 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-185a62881ea84131a0c14987e4296338 received from {username='slave'} at 127.2.248.134:45463
I20240501 08:12:45.880570 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-185a62881ea84131a0c14987e4296338 on tablet 185a62881ea84131a0c14987e4296338 with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:45.888795 14913 ts_tablet_manager.cc:902] T 84789f1228f64be5b6976a9d52f64bd0 P 441e281fafbf42f8938c3f2f42ff7527: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
F20240501 08:12:45.889124 14913 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-185a62881ea84131a0c14987e4296338.queue_time_us, tablet-download-pool-185a62881ea84131a0c14987e4296338.queue_time_us) (tablet-download-pool-185a62881ea84131a0c14987e4296338.run_cpu_time_us, tablet-download-pool-185a62881ea84131a0c14987e4296338.run_cpu_time_us) (tablet-download-pool-185a62881ea84131a0c14987e4296338.run_wall_time_us, tablet-download-pool-185a62881ea84131a0c14987e4296338.run_wall_time_us) (tablet-download-pool-3453d8d883014219b1b14c88e250736b.queue_time_us, tablet-download-pool-3453d8d883014219b1b14c88e250736b.queue_time_us) (tablet-download-pool-3453d8d883014219b1b14c88e250736b.run_cpu_time_us, tablet-download-pool-3453d8d883014219b1b14c88e250736b.run_cpu_time_us) (tablet-download-pool-3453d8d883014219b1b14c88e250736b.run_wall_time_us, tablet-download-pool-3453d8d883014219b1b14c88e250736b.run_wall_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.queue_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.queue_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.run_cpu_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.run_cpu_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.run_wall_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.run_wall_time_us) (tablet-download-pool-6d4ab8e42a484cf89b42f5f53f511265.queue_time_us, tablet-download-pool-6d4ab8e42a484cf89b42f5f53f511265.queue_time_us) (tablet-download-pool-6d4ab8e42a484cf89b42f5f53f511265.run_cpu_time_us, tablet-download-pool-6d4ab8e42a484cf89b42f5f53f511265.run_cpu_time_us) (tablet-download-pool-6d4ab8e42a484cf89b42f5f53f511265.run_wall_time_us, tablet-download-pool-6d4ab8e42a484cf89b42f5f53f511265.run_wall_time_us) (tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.queue_time_us, tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.queue_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_cpu_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_cpu_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_wall_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_wall_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.queue_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.queue_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_cpu_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_cpu_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_wall_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_wall_time_us) (tablet-download-pool-92153666aebe47ddb080278cf24d1570.queue_time_us, tablet-download-pool-92153666aebe47ddb080278cf24d1570.queue_time_us) (tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_cpu_time_us, tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_cpu_time_us) (tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_wall_time_us, tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_wall_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.queue_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.queue_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_cpu_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_cpu_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_wall_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_wall_time_us) (tablet-download-pool-aa6139020ac7458aa8a7ac08f63e6f26.queue_time_us, tablet-download-pool-aa6139020ac7458aa8a7ac08f63e6f26.queue_time_us) (tablet-download-pool-aa6139020ac7458aa8a7ac08f63e6f26.run_cpu_time_us, tablet-download-pool-aa6139020ac7458aa8a7ac08f63e6f26.run_cpu_time_us) (tablet-download-pool-aa6139020ac7458aa8a7ac08f63e6f26.run_wall_time_us, tablet-download-pool-aa6139020ac7458aa8a7ac08f63e6f26.run_wall_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.queue_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.queue_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_cpu_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_cpu_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_wall_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_wall_time_us) (tablet-download-pool-b72323b8b99f4fd1aedcce69089f6204.queue_time_us, tablet-download-pool-b72323b8b99f4fd1aedcce69089f6204.queue_time_us) (tablet-download-pool-b72323b8b99f4fd1aedcce69089f6204.run_cpu_time_us, tablet-download-pool-b72323b8b99f4fd1aedcce69089f6204.run_cpu_time_us) (tablet-download-pool-b72323b8b99f4fd1aedcce69089f6204.run_wall_time_us, tablet-download-pool-b72323b8b99f4fd1aedcce69089f6204.run_wall_time_us) (tablet-download-pool-bbef0147276d410da3d3830e682efe6d.queue_time_us, tablet-download-pool-bbef0147276d410da3d3830e682efe6d.queue_time_us) (tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_cpu_time_us, tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_cpu_time_us) (tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_wall_time_us, tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_wall_time_us) (tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.queue_time_us, tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.queue_time_us) (tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_cpu_time_us, tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_cpu_time_us) (tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_wall_time_us, tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_wall_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.queue_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.queue_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_cpu_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_cpu_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_wall_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 08:12:45.892515 12272 raft_consensus.cc:1203] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
    @     0x7f525999fdcd  google::LogMessage::Fail() at ??:0
    @     0x7f52599a3b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f525999f7cc  google::LogMessage::Flush() at ??:0
    @     0x7f52599a0f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:12:45.906737 14730 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
I20240501 08:12:45.913395 12667 raft_consensus.cc:1203] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240501 08:12:45.997222 10619 consensus_queue.cc:237] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
    @     0x7f525c6e0742  kudu::TraceMetrics::InternName() at ??:0
I20240501 08:12:46.009578 12403 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:46.010351 14862 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:46.012562 10732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:46.019853 10492 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:46.023180 14862 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:46.027439 12402 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:46.031381 12667 raft_consensus.cc:1203] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240501 08:12:46.037668 10557 consensus_peers.cc:485] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea502ae00eff4f72a74b8d25f5a11ede. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:46.037039 10220 catalog_manager.cc:5484] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.042917 14950 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:46.051801 10492 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:46.056232 10487 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:46.067028 12667 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:46.067891 10751 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:46.068457 15054 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:46.070894 15054 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
    @     0x7f525c6acf8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240501 08:12:46.085166 12666 raft_consensus.cc:1203] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240501 08:12:46.110319 14830 raft_consensus.cc:1050] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 to VOTER
I20240501 08:12:46.111356 14830 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
W20240501 08:12:46.113904 10428 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c665d7d20360440a9103e245f5f0900c. This is attempt 1: this message will repeat every 5th retry.
    @     0x7f525c6a6cd6  kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 08:12:46.121002 12516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7f526baf98de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 08:12:46.175269 15054 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:46.180104 12666 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:46.195503 10220 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.234498 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:46.246623 10751 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
    @     0x7f526bb11daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 08:12:46.270426 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:46.273914 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
    @     0x7f526bc3c480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:12:46.301028 14979 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:46.302989 14979 tablet_bootstrap.cc:492] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: Bootstrap complete.
I20240501 08:12:46.303395 14979 ts_tablet_manager.cc:1367] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: Time spent bootstrapping tablet: real 0.791s	user 0.059s	sys 0.005s
I20240501 08:12:46.304375 14979 raft_consensus.cc:348] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:46.304646 14979 raft_consensus.cc:729] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 441e281fafbf42f8938c3f2f42ff7527, State: Initialized, Role: LEARNER
I20240501 08:12:46.304875 14979 consensus_queue.cc:260] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:46.310424 14950 raft_consensus.cc:1050] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:46.311632 15131 ts_tablet_manager.cc:902] T bfbef7af4781448bb04b107b53d58ca0 P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:12:46.311489 14950 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:46.317694 14730 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:46.318168 14730 tablet_bootstrap.cc:492] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap complete.
I20240501 08:12:46.318456 14730 ts_tablet_manager.cc:1367] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Time spent bootstrapping tablet: real 0.412s	user 0.051s	sys 0.016s
I20240501 08:12:46.322113 12536 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.322765 14965 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.325369 14730 raft_consensus.cc:348] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:46.325654 14730 raft_consensus.cc:729] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb2f0099fb794400b5c24927e213e5d8, State: Initialized, Role: LEARNER
I20240501 08:12:46.325879 14730 consensus_queue.cc:260] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
    @     0x7f526bc74a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:12:46.326494 14730 ts_tablet_manager.cc:1398] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.000s
I20240501 08:12:46.327519 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session bb2f0099fb794400b5c24927e213e5d8-6624441a1f1143648913595f905f8039 received from {username='slave'} at 127.2.248.133:47709
I20240501 08:12:46.327661 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session bb2f0099fb794400b5c24927e213e5d8-6624441a1f1143648913595f905f8039 on tablet 6624441a1f1143648913595f905f8039 with peer bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:46.329807 12403 raft_consensus.cc:1203] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 08:12:46.330360 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
    @     0x7f526bc57981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 08:12:46.337101 10487 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.337754 14965 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.352687 14979 ts_tablet_manager.cc:1398] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527: Time spent starting tablet: real 0.049s	user 0.003s	sys 0.000s
I20240501 08:12:46.354837 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session 441e281fafbf42f8938c3f2f42ff7527-cd01dd912e1e4ca9a104270f3cd004b1 received from {username='slave'} at 127.2.248.135:43581
I20240501 08:12:46.355008 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session 441e281fafbf42f8938c3f2f42ff7527-cd01dd912e1e4ca9a104270f3cd004b1 on tablet cd01dd912e1e4ca9a104270f3cd004b1 with peer 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:46.357635 12274 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:46.358852 12535 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:46.359400 14863 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:12:46.361318 14863 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:12:46.368649 14863 raft_consensus.cc:1013] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer bb2f0099fb794400b5c24927e213e5d8: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:12:46.372165 12667 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:46.372795 14950 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:12:46.373150 12535 raft_consensus.cc:1203] T cd01dd912e1e4ca9a104270f3cd004b1 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240501 08:12:46.381752 14862 raft_consensus.cc:1050] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:46.381736 10749 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:12:46.382567 14862 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:46.383858 14863 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
    @     0x7f526bc5770e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 08:12:46.397634 10619 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.398293 14965 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
    @     0x7f525c6b0b75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f525c6b419c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f525c6b4019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 08:12:46.430977 14830 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:46.432231 10492 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:46.434520 12272 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:46.437403 12536 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 441e281fafbf42f8938c3f2f42ff7527 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:46.441329 14950 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:46.442369 10619 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:46.443964 14950 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.450408 10220 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:46.465421 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7f525c68f01e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f525fa146db  start_thread at ??:0
    @     0x7f525885271f  clone at ??:0
I20240501 08:12:46.522171 10487 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:12:46.522787 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:46.523715 12666 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:46.542416 14862 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:46.546541 14830 raft_consensus.cc:1050] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
W20240501 08:12:46.546649 10557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.133:43981, user_credentials={real_user=slave}} blocked reactor thread for 111502us
I20240501 08:12:46.547477 14950 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:46.547480 14830 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:46.549584 12664 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.554283 10749 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.564975 10215 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 11 to 12, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.568354 12272 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.596817 14862 raft_consensus.cc:1050] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER bb2f0099fb794400b5c24927e213e5d8 to VOTER
I20240501 08:12:46.597761 14862 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:46.603524 10749 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:12:46.604383 12664 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:12:46.605357 15150 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:12:46.605839 12272 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:12:46.605993 15163 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:12:46.607172 15163 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:12:46.610198 15155 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.610698 10749 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.618088 10215 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 12 to 13, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.625008 12664 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.629004 12272 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:46.636852 15155 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:46.642432 14730 ts_tablet_manager.cc:902] T ea502ae00eff4f72a74b8d25f5a11ede P bb2f0099fb794400b5c24927e213e5d8: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
F20240501 08:12:46.642791 14730 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.queue_time_us, tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.queue_time_us) (tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_cpu_time_us, tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_cpu_time_us) (tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_wall_time_us, tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_wall_time_us) (tablet-download-pool-2a63e16f11894c2886091dcd3d204106.queue_time_us, tablet-download-pool-2a63e16f11894c2886091dcd3d204106.queue_time_us) (tablet-download-pool-2a63e16f11894c2886091dcd3d204106.run_cpu_time_us, tablet-download-pool-2a63e16f11894c2886091dcd3d204106.run_cpu_time_us) (tablet-download-pool-2a63e16f11894c2886091dcd3d204106.run_wall_time_us, tablet-download-pool-2a63e16f11894c2886091dcd3d204106.run_wall_time_us) (tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.queue_time_us, tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.queue_time_us) (tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_cpu_time_us, tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_cpu_time_us) (tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_wall_time_us, tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_wall_time_us) (tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.queue_time_us, tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.queue_time_us) (tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_cpu_time_us, tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_cpu_time_us) (tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_wall_time_us, tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_wall_time_us) (tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.queue_time_us, tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.queue_time_us) (tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_cpu_time_us, tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_cpu_time_us) (tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_wall_time_us, tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_wall_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.queue_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.queue_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.run_cpu_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.run_cpu_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.run_wall_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.run_wall_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_cpu_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_cpu_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_wall_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_wall_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.queue_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.queue_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_cpu_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_cpu_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_wall_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_wall_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.queue_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.queue_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_cpu_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_cpu_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_wall_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_wall_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.queue_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.queue_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_cpu_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_cpu_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_wall_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_wall_time_us) (tablet-download-pool-ae946c165cd34853917e5c8a244b6408.queue_time_us, tablet-download-pool-ae946c165cd34853917e5c8a244b6408.queue_time_us) (tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_cpu_time_us, tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_cpu_time_us) (tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_wall_time_us, tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_wall_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.queue_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.queue_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_cpu_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_cpu_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_wall_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_wall_time_us) (tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.queue_time_us, tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.queue_time_us) (tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_cpu_time_us, tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_cpu_time_us) (tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_wall_time_us, tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_wall_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.queue_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.queue_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_cpu_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_cpu_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_wall_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_wall_time_us) (tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.queue_time_us, tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 08:12:46.643270 12274 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
    @     0x7f4d6c5c5dcd  google::LogMessage::Fail() at ??:0
I20240501 08:12:46.654302 10215 catalog_manager.cc:5484] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 7 to 8, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
    @     0x7f4d6c5c9b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f4d6c5c57cc  google::LogMessage::Flush() at ??:0
I20240501 08:12:46.663049 15161 raft_consensus.cc:1050] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
    @     0x7f4d6c5c6f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:12:46.696303 12664 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:46.703840 12665 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:46.718552 12254 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:46.739207 15131 tablet_copy_client.cc:308] T bfbef7af4781448bb04b107b53d58ca0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:12:46.755349 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7f4d6f306742  kudu::TraceMetrics::InternName() at ??:0
I20240501 08:12:46.686453 10732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:46.758733 10749 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
    @     0x7f4d6f2d2f8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240501 08:12:46.670634 10492 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:46.694386 10488 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:46.746768 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet bfbef7af4781448bb04b107b53d58ca0 from peer bb2f0099fb794400b5c24927e213e5d8 ({username='slave'} at 127.2.248.133:48051)
I20240501 08:12:46.840045 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet bfbef7af4781448bb04b107b53d58ca0 from peer bb2f0099fb794400b5c24927e213e5d8 at {username='slave'} at 127.2.248.133:48051: session id = bb2f0099fb794400b5c24927e213e5d8-bfbef7af4781448bb04b107b53d58ca0
    @     0x7f4d6f2cccd6  kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 08:12:46.663995 15161 consensus_queue.cc:237] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:46.843358 10512 tablet_copy_source_session.cc:205] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:46.844727 15131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbef7af4781448bb04b107b53d58ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:46.849128 12665 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 08:12:46.850775 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a6ed4ea6d6941d2af0f81eaf744bcea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:46.851802 12272 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:46.852149 15131 tablet_copy_client.cc:783] T bfbef7af4781448bb04b107b53d58ca0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:46.852380 15131 tablet_copy_client.cc:647] T bfbef7af4781448bb04b107b53d58ca0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:46.852421 14723 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:46.853809 15161 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:46.854494 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
    @     0x7f4d7e71f8de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f4d7e737daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f4d7e862480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f4d7e89aa8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:12:46.939321 14723 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:12:47.014628 10688 connection.cc:535] client connection to 127.2.248.135:37749 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:12:47.128466 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
    @     0x7f4d7e87d981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f4d7e87d70e  std::_Function_handler<>::_M_invoke() at ??:0
I20240501 08:12:46.872071 15131 tablet_copy_client.cc:523] T bfbef7af4781448bb04b107b53d58ca0 P bb2f0099fb794400b5c24927e213e5d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:46.717432 10358 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:46.934721 15208 raft_consensus.cc:1050] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea to VOTER
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 08:12:47.186378 15208 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:46.690503 12403 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:46.864238 10621 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:46.940994 12272 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:46.749960 14830 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:47.191035 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:12:47.005110 10428 connection.cc:535] client connection to 127.2.248.135:37749 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:12:47.192353 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:46.864609 10619 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:12:47.196215 14830 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:46.852690 10359 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:47.197479 14375 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:12:47.005599 10558 connection.cc:535] client connection to 127.2.248.135:37749 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:12:47.199162 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:46.898847 10361 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 08:12:47.005369 10298 connection.cc:535] client connection to 127.2.248.135:37749 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:12:47.202284 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:47.175972 10362 consensus_queue.cc:237] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:47.207525 12664 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:12:47.208020 12667 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:47.208271 15155 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:47.210405 14723 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:47.213726 15164 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:12:47.214309 15164 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:47.215970 10749 raft_consensus.cc:1261] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.219862 14965 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.220976 10619 raft_consensus.cc:1261] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.227473 12274 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 08:12:47.227780 12273 raft_consensus.cc:1261] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:47.228340 15131 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P bb2f0099fb794400b5c24927e213e5d8: Bootstrap starting.
W20240501 08:12:47.230038 10297 consensus_peers.cc:485] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e99538651cf64510a811ba01c6c8fb40. This is attempt 1: this message will repeat every 5th retry.
    @     0x7f4d6f2d6b75  kudu::ThreadPool::DispatchThread() at ??:0
I20240501 08:12:47.231328 15163 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:12:47.231832 15163 consensus_queue.cc:1035] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:47.232247 15150 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:47.238413 10215 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P bb2f0099fb794400b5c24927e213e5d8 reported cstate change: config changed from index 7 to 8, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:12:47.241907 14375 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.248308 12667 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:47.248637 12273 raft_consensus.cc:2932] T 03d0f89e1d8f40fe84d275b117d8145f P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
I20240501 08:12:47.253329 14830 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
    @     0x7f4d6f2da19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 08:12:47.256086 15161 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.256898 10749 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:47.257191 15155 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.259250 12272 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P bb2f0099fb794400b5c24927e213e5d8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.259940 10362 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.272348 10361 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
    @     0x7f4d6f2da019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 08:12:47.281271 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03d0f89e1d8f40fe84d275b117d8145f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.283993 10732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:47.287566 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6624441a1f1143648913595f905f8039 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.288985 12664 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
W20240501 08:12:47.297017 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.297163 10688 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.297292 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
    @     0x7f4d6f2b501e  kudu::Thread::SuperviseThread() at ??:0
I20240501 08:12:47.310899 12664 raft_consensus.cc:2932] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
    @     0x7f4d7263a6db  start_thread at ??:0
    @     0x7f4d6b47871f  clone at ??:0
I20240501 08:12:47.328240 10217 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 13 to 14, VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.329020 10217 catalog_manager.cc:5484] T 03d0f89e1d8f40fe84d275b117d8145f P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 8 to 9, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } }
W20240501 08:12:47.351372 10298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 149121us
W20240501 08:12:47.352645 10428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 160429us
I20240501 08:12:47.355664 10751 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:47.356403 12665 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:47.356417 14965 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.357709 12403 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:47.358366 15054 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:47.360493 14375 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:47.361266 10749 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:12:47.361876 15054 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:12:47.364832 12665 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.365739 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:47.445101 15285 ts_tablet_manager.cc:902] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
W20240501 08:12:47.383234 10558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 184176us
I20240501 08:12:47.488566 10732 tablet_service.cc:1514] Processing DeleteTablet for tablet 6624441a1f1143648913595f905f8039 with delete_type TABLET_DATA_TOMBSTONED (TS 06c1439c9ab8415485eff20566811ede not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:41424
I20240501 08:12:47.476797 10220 catalog_manager.cc:5484] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.365397 12402 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.541821 15307 tablet_replica.cc:330] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: stopping tablet replica
I20240501 08:12:47.509951 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:47.542052 15307 raft_consensus.cc:2229] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.542316 15307 raft_consensus.cc:2258] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.542214 10220 catalog_manager.cc:5484] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 7 to 8, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.427784 15205 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
W20240501 08:12:47.474869 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.543702 10428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 70154.8us
W20240501 08:12:47.543926 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:47.365778 14830 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:47.392639 10619 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
W20240501 08:12:47.413126 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.544976 10298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 131854us
W20240501 08:12:47.545240 10298 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.429185 12403 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:47.486452 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d0f89e1d8f40fe84d275b117d8145f with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:47.567062 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:47.651685 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:12:47.560010 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.637427 15311 tablet_replica.cc:330] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
W20240501 08:12:47.660372 10428 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 100340us
I20240501 08:12:47.660498 15311 raft_consensus.cc:2229] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.660794 15311 raft_consensus.cc:2258] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.561291 12664 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:47.554064 14375 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:47.437800 10341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:12:47.567785 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.664593 10298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 96842.2us
I20240501 08:12:47.550598 10751 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:12:47.671162 10492 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.552824 15307 ts_tablet_manager.cc:1875] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.652706 10749 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:47.674191 15311 ts_tablet_manager.cc:1875] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.679980 15307 ts_tablet_manager.cc:1888] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240501 08:12:47.680107 15307 log.cc:1196] T 6624441a1f1143648913595f905f8039 P 06c1439c9ab8415485eff20566811ede: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/6624441a1f1143648913595f905f8039
I20240501 08:12:47.691574 10472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:12:47.716733 10427 connection.cc:535] client connection to 127.2.248.133:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:12:47.717213 10297 connection.cc:535] client connection to 127.2.248.133:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:47.721575 15311 ts_tablet_manager.cc:1888] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:12:47.746397 15311 log.cc:1196] T 03d0f89e1d8f40fe84d275b117d8145f P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/03d0f89e1d8f40fe84d275b117d8145f
W20240501 08:12:47.746726 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.747856 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:12:47.728909 15285 tablet_copy_client.cc:308] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
W20240501 08:12:47.655311 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:47.716997 10557 connection.cc:535] client connection to 127.2.248.133:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:12:47.768167 10558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.2.248.135:37749, user_credentials={real_user=slave}} blocked reactor thread for 112873us
W20240501 08:12:47.768507 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.768766 10558 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.771740 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:47.771859 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.776788 10558 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.777392 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:12:47.716720 10687 connection.cc:535] client connection to 127.2.248.133:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:47.770017 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:44587)
I20240501 08:12:47.797312 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 8a6ed4ea6d6941d2af0f81eaf744bcea from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:44587: session id = c9f609481ded4d1aad9dd5f2341fd931-8a6ed4ea6d6941d2af0f81eaf744bcea
I20240501 08:12:47.557168 10217 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 9 to 10, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.559577 10215 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:47.681609 10207 catalog_manager.cc:4830] TS 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): tablet 6624441a1f1143648913595f905f8039 (table rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]) successfully deleted
W20240501 08:12:47.799048 10207 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 117504us
W20240501 08:12:47.799600 10207 connection.cc:535] server connection from 127.2.248.133:44895 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:12:47.682523 10221 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:47.800146 10772 tablet_copy_source_session.cc:205] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.747352 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet 03d0f89e1d8f40fe84d275b117d8145f (table rebalance_test_table_7 [id=e005ad4971044cda8ed96708209e8b34]) successfully deleted
W20240501 08:12:47.800967 10205 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.2.248.130:46605, user_credentials={real_user=slave}} blocked reactor thread for 53645.2us
W20240501 08:12:47.801762 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.801892 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.801990 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:47.802068 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.802527 15285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.809055 10621 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:47.816370 10619 consensus_queue.cc:237] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:47.817178 10358 consensus_queue.cc:237] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:47.819993 12403 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:47.820363 12664 raft_consensus.cc:1261] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 08:12:47.821226 14375 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:47.822126 12664 raft_consensus.cc:1261] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:12:47.822427 15163 consensus_queue.cc:1035] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:12:47.823837 14830 consensus_queue.cc:1035] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:12:47.825978 15161 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.828655 15278 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.829238 12402 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.834376 12665 raft_consensus.cc:2932] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.835438 12664 raft_consensus.cc:2932] T d188d415f63f448195ae90e8e3d66e0e P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.843371 10221 catalog_manager.cc:5484] T 6624441a1f1143648913595f905f8039 P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 14 to 15, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.846658 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6624441a1f1143648913595f905f8039 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.853080 15285 tablet_copy_client.cc:783] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:47.853578 15285 tablet_copy_client.cc:647] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:47.856060 15285 tablet_copy_client.cc:523] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:47.856467 10212 catalog_manager.cc:5484] T d188d415f63f448195ae90e8e3d66e0e P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 10 to 11, VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:47.858121 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d188d415f63f448195ae90e8e3d66e0e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.862062 15285 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
W20240501 08:12:47.866015 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:47.869612 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet d188d415f63f448195ae90e8e3d66e0e with delete_type TABLET_DATA_TOMBSTONED (TS 861f629e2d134ff08f25fa553dfb5bda not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44486
I20240501 08:12:47.870126 15311 tablet_replica.cc:330] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: stopping tablet replica
I20240501 08:12:47.870427 15311 raft_consensus.cc:2229] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.870723 15311 raft_consensus.cc:2258] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.872807 15311 ts_tablet_manager.cc:1875] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:12:47.881374 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:47.888226 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.894743 15311 ts_tablet_manager.cc:1888] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 08:12:47.895442 15311 log.cc:1196] T d188d415f63f448195ae90e8e3d66e0e P 861f629e2d134ff08f25fa553dfb5bda: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/d188d415f63f448195ae90e8e3d66e0e
I20240501 08:12:47.896672 10205 catalog_manager.cc:4830] TS 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): tablet d188d415f63f448195ae90e8e3d66e0e (table rebalance_test_table_6 [id=975dda02343941d7a896ed6459039ea6]) successfully deleted
W20240501 08:12:47.915058 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.917146 10751 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:47.918257 15163 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:12:47.932621 12402 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:12:47.933522 15163 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:12:47.954185 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.955502 15163 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:47.960391 10749 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:47.975538 10215 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
I20240501 08:12:47.979311 12402 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } }
W20240501 08:12:47.993801 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:48.007755 15285 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.010774 15285 tablet_bootstrap.cc:492] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:12:48.012630 10358 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265"
dest_uuid: "c3747ec2a93c4684962f215616e896ed"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53072
I20240501 08:12:48.012908 10358 raft_consensus.cc:595] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:48.025370 15285 ts_tablet_manager.cc:1367] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.163s	user 0.064s	sys 0.000s
I20240501 08:12:48.027158 15285 raft_consensus.cc:348] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:48.028952 15285 raft_consensus.cc:729] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:12:48.029281 15285 consensus_queue.cc:260] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:12:48.035267 15285 ts_tablet_manager.cc:1398] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240501 08:12:48.037413 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-8a6ed4ea6d6941d2af0f81eaf744bcea received from {username='slave'} at 127.2.248.136:44587
I20240501 08:12:48.037597 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-8a6ed4ea6d6941d2af0f81eaf744bcea on tablet 8a6ed4ea6d6941d2af0f81eaf744bcea with peer c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:48.061100 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:48.082943 10358 consensus_queue.cc:237] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:48.091342 12402 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:12:48.091845 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.091966 10492 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:48.092657 15278 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:48.093006 15283 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:48.096103 15278 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.097558 12402 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.099886 10492 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.105360 10215 catalog_manager.cc:5484] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.178717 10751 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
I20240501 08:12:48.182129 12403 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:48.184494 10621 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:48.185953 14359 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:48.186551 14359 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:12:48.194355 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.200894 15219 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.205035 10621 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.214754 12403 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.223093 10215 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 8 to 9, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:48.252488 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:48.254362 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:12:48.283890 10492 consensus_queue.cc:237] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } }
W20240501 08:12:48.294143 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:48.295804 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.296842 10358 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:12:48.297434 15269 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:48.304544 14830 raft_consensus.cc:979] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed: : Instructing follower 861f629e2d134ff08f25fa553dfb5bda to start an election
I20240501 08:12:48.304708 14830 raft_consensus.cc:1067] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Signalling peer 861f629e2d134ff08f25fa553dfb5bda to start an election
I20240501 08:12:48.305724 10488 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265"
dest_uuid: "861f629e2d134ff08f25fa553dfb5bda"
 from {username='slave'} at 127.2.248.129:48311
I20240501 08:12:48.306013 10488 raft_consensus.cc:482] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:48.306162 10488 raft_consensus.cc:3037] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:48.309614 10488 raft_consensus.cc:504] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:48.310961 10621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:48.310961 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ab8e42a484cf89b42f5f53f511265" candidate_uuid: "861f629e2d134ff08f25fa553dfb5bda" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:48.311188 10621 raft_consensus.cc:3037] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:48.311266 10362 raft_consensus.cc:3032] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:48.311363 10362 raft_consensus.cc:729] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:12:48.311779 10362 consensus_queue.cc:260] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:48.312494 10362 raft_consensus.cc:3037] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:48.314535 10621 raft_consensus.cc:2454] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 2.
I20240501 08:12:48.315339 10488 leader_election.cc:288] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 2 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749)
W20240501 08:12:48.317545 10428 leader_election.cc:334] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:48.318521 10362 raft_consensus.cc:2454] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861f629e2d134ff08f25fa553dfb5bda in term 2.
I20240501 08:12:48.319017 10427 leader_election.cc:302] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 861f629e2d134ff08f25fa553dfb5bda, 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed; no voters: 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:48.321668 15269 raft_consensus.cc:2781] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:48.321846 15269 raft_consensus.cc:686] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 2 LEADER]: Becoming Leader. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Running, Role: LEADER
I20240501 08:12:48.322185 15269 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
I20240501 08:12:48.341610 12715 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3vHYYh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 08:12:30.047820 12720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:30.048473 12720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:30.064332 12720 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:33.395013 12720 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:12:35.415930 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.416343 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:35.416479 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.416821 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.416941 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.417097 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:35.417193 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.417308 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:35.417426 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:35.417577 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:35.417714 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:35.417825 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:35.417922 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:35.418037 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:35.418133 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:35.585409 12720 rebalancer_tool.cc:1074] tablet d188d415f63f448195ae90e8e3d66e0e: '06c1439c9ab8415485eff20566811ede' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:35.751544 12720 rebalancer_tool.cc:1074] tablet c665d7d20360440a9103e245f5f0900c: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:36.039404 12720 rebalancer_tool.cc:1074] tablet 50b7150bbeb645d1b00d1a561c3276d1: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:36.633087 12720 rebalancer_tool.cc:1074] tablet cf1f28d15c094fd98fac1b91d2785744: '861f629e2d134ff08f25fa553dfb5bda' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:36.842810 12720 rebalancer_tool.cc:1084] tablet 92153666aebe47ddb080278cf24d1570: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move ignored: Illegal state: tablet 92153666aebe47ddb080278cf24d1570: replica b6a964e54f3b44eda6d0666e0c1e4bea is already present
I20240501 08:12:36.842952 12720 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:40.959815 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.978899 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:40.979034 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:40.979413 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:40.979512 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 441e281fafbf42f8938c3f2f42ff7527 to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.979619 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:40.979797 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.979923 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:40.980051 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:40.980167 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.980301 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:40.980475 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.980583 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:40.980665 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:41.210536 12720 rebalancer_tool.cc:1074] tablet af24b8ab4d0c43ef986d89d4aefc263c: '06c1439c9ab8415485eff20566811ede' -> '' move scheduled
I20240501 08:12:41.299038 12720 rebalancer_tool.cc:1084] tablet 4b924c6344714a92ac37ae28961b02f5: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move ignored: Illegal state: tablet 4b924c6344714a92ac37ae28961b02f5: replica bb2f0099fb794400b5c24927e213e5d8 is already present
I20240501 08:12:41.299196 12720 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:43.696504 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:43.697034 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:43.697194 12720 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:43.805374 12720 rebalancer_tool.cc:1074] tablet 21065b7dcc18456eb1b4c2376d51eec4: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:44.009441 12720 rebalancer_tool.cc:1074] tablet 03d0f89e1d8f40fe84d275b117d8145f: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:44.418969 12720 rebalancer_tool.cc:1074] tablet cd01dd912e1e4ca9a104270f3cd004b1: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '' move scheduled
I20240501 08:12:44.795217 12720 rebalancer_tool.cc:1074] tablet 6624441a1f1143648913595f905f8039: '06c1439c9ab8415485eff20566811ede' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:45.100019 12720 rebalancer_tool.cc:1074] tablet 8a6ed4ea6d6941d2af0f81eaf744bcea: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:45.316800 12720 rebalancer_tool.cc:1084] tablet 84789f1228f64be5b6976a9d52f64bd0: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:45.316928 12720 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:47.017352 15182 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:12:46.578781 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.135:37749 (local address 127.0.0.1:39846)
0501 08:12:46.594115 (+ 15334us) negotiation.cc:107] Waiting for socket to connect
0501 08:12:46.594135 (+    20us) client_negotiation.cc:174] Beginning negotiation
0501 08:12:46.594255 (+   120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:12:47.015774 (+421519us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15209,"thread_start_us":59574,"threads_started":1}
Illegal state: tablet server 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): unacceptable health status UNAVAILABLE

I20240501 08:12:48.344614 15418 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.345422 10362 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.367235 10212 catalog_manager.cc:5484] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 6 to 7, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: true } } }
I20240501 08:12:48.378156 10212 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: term changed from 1 to 2, leader changed from c3747ec2a93c4684962f215616e896ed (127.2.248.129) to 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130). New cstate: current_term: 2 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:48.507483 12664 raft_consensus.cc:1203] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20240501 08:12:48.521203 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:48.552603 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:48.562402 12717 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3vHYYh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 08:12:29.674171 12721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:29.674616 12721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:29.689291 12721 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:32.913856 12721 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:12:34.597412 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.597625 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.597770 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.597935 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.598093 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.598287 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.598510 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.598626 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.598727 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.598825 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.598955 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.599112 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.599224 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.599370 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.599467 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:34.895998 12721 rebalancer_tool.cc:1074] tablet 8f27eb42e75247c2aa142d3028e6ca2a: 'c3747ec2a93c4684962f215616e896ed' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:35.079152 12721 rebalancer_tool.cc:1074] tablet d188d415f63f448195ae90e8e3d66e0e: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:35.280867 12721 rebalancer_tool.cc:1074] tablet ae946c165cd34853917e5c8a244b6408: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:35.431998 12721 rebalancer_tool.cc:1074] tablet bbef0147276d410da3d3830e682efe6d: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:35.513990 12721 rebalancer_tool.cc:1074] tablet bfbef7af4781448bb04b107b53d58ca0: 'c3747ec2a93c4684962f215616e896ed' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:35.694180 12721 rebalancer_tool.cc:1074] tablet af24b8ab4d0c43ef986d89d4aefc263c: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:35.916632 12721 rebalancer_tool.cc:1074] tablet c665d7d20360440a9103e245f5f0900c: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:36.251807 12721 rebalancer_tool.cc:1074] tablet 6624441a1f1143648913595f905f8039: '861f629e2d134ff08f25fa553dfb5bda' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:36.662027 12721 rebalancer_tool.cc:1074] tablet 185a62881ea84131a0c14987e4296338: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:37.145360 12721 rebalancer_tool.cc:1074] tablet ea502ae00eff4f72a74b8d25f5a11ede: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:37.476058 12721 rebalancer_tool.cc:1074] tablet 850d4e1020e243f4b6fa113d2946846d: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:37.715497 12721 rebalancer_tool.cc:1084] tablet 50b7150bbeb645d1b00d1a561c3276d1: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move ignored: Illegal state: tablet 50b7150bbeb645d1b00d1a561c3276d1: replica bb2f0099fb794400b5c24927e213e5d8 is already present
I20240501 08:12:37.715688 12721 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:40.970131 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:40.970317 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.970399 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 441e281fafbf42f8938c3f2f42ff7527 to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:40.970707 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:40.970829 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:40.970930 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:40.971067 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.971218 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:40.971321 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.971410 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:40.971488 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:41.123068 12721 rebalancer_tool.cc:1074] tablet 4b924c6344714a92ac37ae28961b02f5: '861f629e2d134ff08f25fa553dfb5bda' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:41.374190 12721 rebalancer_tool.cc:1074] tablet 2a63e16f11894c2886091dcd3d204106: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:41.686012 12721 rebalancer_tool.cc:1074] tablet 3453d8d883014219b1b14c88e250736b: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:42.001120 12721 rebalancer_tool.cc:1074] tablet cf1f28d15c094fd98fac1b91d2785744: '06c1439c9ab8415485eff20566811ede' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:42.283275 12721 rebalancer_tool.cc:1074] tablet 50b7150bbeb645d1b00d1a561c3276d1: '861f629e2d134ff08f25fa553dfb5bda' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:42.555159 12721 rebalancer_tool.cc:1074] tablet 4799a93d96a64a75b1984bb85fcad7e8: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:42.639927 12721 rebalancer_tool.cc:1084] tablet f5a58b7166534844b0963f4fab46dfd9: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move ignored: Illegal state: tablet f5a58b7166534844b0963f4fab46dfd9: replica b6a964e54f3b44eda6d0666e0c1e4bea is already present
I20240501 08:12:42.640062 12721 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:44.675597 14783 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.2.248.190:33713, user_credentials={real_user=slave}} blocked reactor thread for 52873us
W20240501 08:12:45.925491 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:45.925715 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:45.925889 12721 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:46.095022 12721 rebalancer_tool.cc:1084] tablet 84789f1228f64be5b6976a9d52f64bd0: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move ignored: Illegal state: tablet 84789f1228f64be5b6976a9d52f64bd0: replica 441e281fafbf42f8938c3f2f42ff7527 is already present
I20240501 08:12:46.095208 12721 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:47.719096 15257 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:12:47.396861 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.133:43981 (local address 127.0.0.1:53248)
0501 08:12:47.400062 (+  3201us) negotiation.cc:107] Waiting for socket to connect
0501 08:12:47.400082 (+    20us) client_negotiation.cc:174] Beginning negotiation
0501 08:12:47.400230 (+   148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:12:47.717622 (+317392us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3074}
Illegal state: tablet server 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): unacceptable health status UNAVAILABLE

I20240501 08:12:48.568220 10751 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1"
dest_uuid: "06c1439c9ab8415485eff20566811ede"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35254
I20240501 08:12:48.568611 10751 raft_consensus.cc:595] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:48.602514 10732 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:48.603713 10600 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:48.640825 10472 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:12:48.647545 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:12:48.714917 12385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:48.720052 10341 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:12:48.760267 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:12:48.764191 10621 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:48.764844 15269 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:48.767239 12647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:12:48.795951 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.869158 14723 raft_consensus.cc:979] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede: : Instructing follower b6a964e54f3b44eda6d0666e0c1e4bea to start an election
I20240501 08:12:48.869330 14723 raft_consensus.cc:1067] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Signalling peer b6a964e54f3b44eda6d0666e0c1e4bea to start an election
I20240501 08:12:48.870636 12403 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1"
dest_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea"
 from {username='slave'} at 127.2.248.132:54231
I20240501 08:12:48.870985 12403 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:48.871201 12403 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:48.875838 12403 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:48.885596 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:48.885882 10362 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:48.896601 10362 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a964e54f3b44eda6d0666e0c1e4bea in term 2.
I20240501 08:12:48.897466 12340 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a964e54f3b44eda6d0666e0c1e4bea, c3747ec2a93c4684962f215616e896ed; no voters: 
I20240501 08:12:48.905437 12403 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [CANDIDATE]: Term 2 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:12:48.918622 15516 raft_consensus.cc:2781] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:48.919526 10751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:12:48.919834 10751 raft_consensus.cc:3032] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:48.919952 10751 raft_consensus.cc:729] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Running, Role: LEADER
I20240501 08:12:48.920398 10751 consensus_queue.cc:260] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:12:48.921104 10751 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:48.925026 15219 raft_consensus.cc:1050] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
I20240501 08:12:48.925968 15219 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:48.930476 10362 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:48.931275 15269 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:48.938735 10751 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a964e54f3b44eda6d0666e0c1e4bea in term 2.
I20240501 08:12:48.955798 12664 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:48.956944 10358 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:12:48.956966 14723 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:48.957999 14723 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:12:48.959585 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:48.975685 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.978796 14723 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:48.990784 10362 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:48.995457 10215 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 8 to 9, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:48.985193 15516 raft_consensus.cc:686] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [term 2 LEADER]: Becoming Leader. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Running, Role: LEADER
I20240501 08:12:49.000867 12664 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:12:49.001842 15516 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
W20240501 08:12:49.012166 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:49.018399 10221 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: term changed from 1 to 2, leader changed from 06c1439c9ab8415485eff20566811ede (127.2.248.132) to b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134). New cstate: current_term: 2 leader_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:49.129421 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:49.131402 10619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d"
dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47916
I20240501 08:12:49.131640 10619 raft_consensus.cc:595] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Received request to transfer leadership
W20240501 08:12:49.149755 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:49.216984 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:12:49.257889 12718 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3vHYYh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 08:12:29.936491 12724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:29.936865 12724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:29.955933 12724 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:32.980262 12724 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:12:34.628186 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.628775 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.628994 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.629266 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.629393 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.629575 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.629710 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.629885 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.630055 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.630193 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.630379 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.630556 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.630703 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.630856 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.631017 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
I20240501 08:12:35.096560 12724 rebalancer_tool.cc:1074] tablet af24b8ab4d0c43ef986d89d4aefc263c: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:35.206266 12724 rebalancer_tool.cc:1074] tablet c665d7d20360440a9103e245f5f0900c: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:35.398636 12724 rebalancer_tool.cc:1074] tablet 8f27eb42e75247c2aa142d3028e6ca2a: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:35.529068 12724 rebalancer_tool.cc:1074] tablet 6624441a1f1143648913595f905f8039: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:35.680900 12724 rebalancer_tool.cc:1074] tablet 03d0f89e1d8f40fe84d275b117d8145f: 'c3747ec2a93c4684962f215616e896ed' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:35.938048 12724 rebalancer_tool.cc:1074] tablet 8a6ed4ea6d6941d2af0f81eaf744bcea: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:36.313357 12724 rebalancer_tool.cc:1074] tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:36.834201 12724 rebalancer_tool.cc:1074] tablet 92153666aebe47ddb080278cf24d1570: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:37.176076 12724 rebalancer_tool.cc:1074] tablet 9b6f09b0b5e84329944360d511578ed0: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:37.506853 12724 rebalancer_tool.cc:1084] tablet cd01dd912e1e4ca9a104270f3cd004b1: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:12:37.507006 12724 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:41.044557 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:41.044739 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:41.044852 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:41.045542 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:41.045670 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:41.045830 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:41.045925 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:41.051556 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:41.051661 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:41.051779 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:41.051889 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:41.052017 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:41.186017 12724 rebalancer_tool.cc:1074] tablet 4b924c6344714a92ac37ae28961b02f5: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:41.498688 12724 rebalancer_tool.cc:1074] tablet f5a58b7166534844b0963f4fab46dfd9: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:41.784615 12724 rebalancer_tool.cc:1074] tablet 4799a93d96a64a75b1984bb85fcad7e8: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:42.412278 12724 rebalancer_tool.cc:1074] tablet 21065b7dcc18456eb1b4c2376d51eec4: 'c3747ec2a93c4684962f215616e896ed' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:42.493606 12724 rebalancer_tool.cc:1084] tablet 3453d8d883014219b1b14c88e250736b: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move ignored: Illegal state: tablet 3453d8d883014219b1b14c88e250736b: replica 441e281fafbf42f8938c3f2f42ff7527 is already present
I20240501 08:12:42.493793 12724 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:45.766093 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:45.770139 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:45.770314 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:45.770500 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:45.770653 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:45.770798 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:45.770956 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:45.771234 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:45.771395 12724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server bb2f0099fb794400b5c24927e213e5d8 to server c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:46.029806 12724 rebalancer_tool.cc:1074] tablet ea502ae00eff4f72a74b8d25f5a11ede: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:47.693590 12724 rebalancer_tool.cc:1074] tablet ae946c165cd34853917e5c8a244b6408: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:47.851781 12724 rebalancer_tool.cc:1084] tablet d188d415f63f448195ae90e8e3d66e0e: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move ignored: Illegal state: Request specified cas_config_opid_index of 10 but the committed config has opid_index of 11
I20240501 08:12:47.851934 12724 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): unacceptable health status UNAVAILABLE

I20240501 08:12:49.267175 15532 raft_consensus.cc:979] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf: : Instructing follower c9f609481ded4d1aad9dd5f2341fd931 to start an election
I20240501 08:12:49.267360 15532 raft_consensus.cc:1067] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Signalling peer c9f609481ded4d1aad9dd5f2341fd931 to start an election
I20240501 08:12:49.268955 12664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d"
dest_uuid: "c9f609481ded4d1aad9dd5f2341fd931"
 from {username='slave'} at 127.2.248.131:48871
I20240501 08:12:49.269245 12664 raft_consensus.cc:482] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:49.269426 12664 raft_consensus.cc:3037] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.272413 12664 raft_consensus.cc:504] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:49.275094 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "c9f609481ded4d1aad9dd5f2341fd931" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c3747ec2a93c4684962f215616e896ed"
I20240501 08:12:49.275353 10362 raft_consensus.cc:3037] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.276613 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "c9f609481ded4d1aad9dd5f2341fd931" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf"
I20240501 08:12:49.277112 10619 raft_consensus.cc:3032] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:49.277222 10619 raft_consensus.cc:729] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Running, Role: LEADER
I20240501 08:12:49.277567 10619 consensus_queue.cc:260] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:49.277971 10362 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f609481ded4d1aad9dd5f2341fd931 in term 2.
I20240501 08:12:49.278035 10619 raft_consensus.cc:3037] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.282514 10619 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f609481ded4d1aad9dd5f2341fd931 in term 2.
I20240501 08:12:49.283164 12600 leader_election.cc:302] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf, c3747ec2a93c4684962f215616e896ed, c9f609481ded4d1aad9dd5f2341fd931; no voters: 
I20240501 08:12:49.283550 15264 raft_consensus.cc:2781] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:49.283988 15264 raft_consensus.cc:686] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [term 2 LEADER]: Becoming Leader. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Running, Role: LEADER
I20240501 08:12:49.284521 15264 consensus_queue.cc:237] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:12:49.289930 12664 leader_election.cc:288] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [CANDIDATE]: Term 2 election: Requested vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749)
W20240501 08:12:49.295009 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:49.295706 10221 catalog_manager.cc:5484] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: term changed from 1 to 2, leader changed from 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) to c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136). New cstate: current_term: 2 leader_uuid: "c9f609481ded4d1aad9dd5f2341fd931" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:12:49.305599 12603 leader_election.cc:334] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:49.360597 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:49.371711 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:49.396497 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:12:49.398569 10732 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:12:49.406265 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:49.408708 10472 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:12:49.424975 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:49.454098 10751 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Refusing update from remote peer b6a964e54f3b44eda6d0666e0c1e4bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:12:49.455013 15516 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:12:49.482587 10600 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:49.486946 12385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:49.491714 10341 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:49.496809 12647 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:12:49.570734 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:49.573680 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:12:49.574820 10362 raft_consensus.cc:1261] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Refusing update from remote peer b6a964e54f3b44eda6d0666e0c1e4bea: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:12:49.575413 15516 consensus_queue.cc:1035] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:12:49.630970 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:49.689925 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:12:49.691916 10362 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Refusing update from remote peer c9f609481ded4d1aad9dd5f2341fd931: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:12:49.692591 15264 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:12:49.728983 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:49.744462 12716 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3vHYYh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 08:12:29.684954 12723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:29.685514 12723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:29.698791 12723 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:32.841233 12723 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:12:34.852985 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.853293 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.853427 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.853566 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.853773 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.853924 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.854017 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.854127 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:34.854252 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.854393 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.854488 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:34.854635 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:34.854777 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:34.854892 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:35.175192 12723 rebalancer_tool.cc:1074] tablet cf1f28d15c094fd98fac1b91d2785744: '861f629e2d134ff08f25fa553dfb5bda' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:35.362586 12723 rebalancer_tool.cc:1074] tablet b72323b8b99f4fd1aedcce69089f6204: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:35.601888 12723 rebalancer_tool.cc:1074] tablet 8f27eb42e75247c2aa142d3028e6ca2a: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:35.763605 12723 rebalancer_tool.cc:1074] tablet ae946c165cd34853917e5c8a244b6408: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:35.997916 12723 rebalancer_tool.cc:1074] tablet 92153666aebe47ddb080278cf24d1570: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:36.308169 12723 rebalancer_tool.cc:1074] tablet aa6139020ac7458aa8a7ac08f63e6f26: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:36.571857 12723 rebalancer_tool.cc:1074] tablet 9b6f09b0b5e84329944360d511578ed0: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:36.967669 12723 rebalancer_tool.cc:1074] tablet 850d4e1020e243f4b6fa113d2946846d: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:37.554257 12723 rebalancer_tool.cc:1074] tablet cd01dd912e1e4ca9a104270f3cd004b1: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:37.840050 12723 rebalancer_tool.cc:1084] tablet 6624441a1f1143648913595f905f8039: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move ignored: Illegal state: tablet 6624441a1f1143648913595f905f8039: replica 441e281fafbf42f8938c3f2f42ff7527 is already present
I20240501 08:12:37.840799 12723 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:41.130651 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:41.130841 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:41.155984 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:41.156301 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:41.156467 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:41.156580 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:41.156684 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:41.156890 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:41.335783 12723 rebalancer_tool.cc:1074] tablet af24b8ab4d0c43ef986d89d4aefc263c: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '' move scheduled
I20240501 08:12:41.512265 12723 rebalancer_tool.cc:1084] tablet 4b924c6344714a92ac37ae28961b02f5: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move ignored: Illegal state: tablet 4b924c6344714a92ac37ae28961b02f5: replica bb2f0099fb794400b5c24927e213e5d8 is already present
I20240501 08:12:41.513046 12723 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:42.454901 14563 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.2.248.190:33713, user_credentials={real_user=slave}} blocked reactor thread for 55991.9us
W20240501 08:12:43.692323 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:43.692709 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:43.692845 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:43.692957 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:43.693037 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:43.693152 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:43.693238 12723 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:44.021629 12723 rebalancer_tool.cc:1074] tablet 6d4ab8e42a484cf89b42f5f53f511265: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:44.199972 12723 rebalancer_tool.cc:1074] tablet 6624441a1f1143648913595f905f8039: '06c1439c9ab8415485eff20566811ede' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:44.743520 12723 rebalancer_tool.cc:1074] tablet d188d415f63f448195ae90e8e3d66e0e: '861f629e2d134ff08f25fa553dfb5bda' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:45.382771 12723 rebalancer_tool.cc:1074] tablet 84789f1228f64be5b6976a9d52f64bd0: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:45.799399 12723 rebalancer_tool.cc:1074] tablet bfbef7af4781448bb04b107b53d58ca0: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:46.186825 12723 rebalancer_tool.cc:1074] tablet c665d7d20360440a9103e245f5f0900c: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:47.130565 12723 rebalancer_tool.cc:1074] tablet 8a6ed4ea6d6941d2af0f81eaf744bcea: 'c3747ec2a93c4684962f215616e896ed' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:47.517498 12723 rebalancer_tool.cc:1136] tablet aa6139020ac7458aa8a7ac08f63e6f26: '861f629e2d134ff08f25fa553dfb5bda' -> '441e281fafbf42f8938c3f2f42ff7527' move completed: OK
I20240501 08:12:47.532927 12723 rebalancer_tool.cc:1136] tablet 92153666aebe47ddb080278cf24d1570: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move completed: OK
I20240501 08:12:47.710059 12723 rebalancer_tool.cc:1136] tablet 9b6f09b0b5e84329944360d511578ed0: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move completed: OK
I20240501 08:12:48.163393 12723 rebalancer_tool.cc:1136] tablet d188d415f63f448195ae90e8e3d66e0e: '861f629e2d134ff08f25fa553dfb5bda' -> 'c9f609481ded4d1aad9dd5f2341fd931' move completed: OK
I20240501 08:12:48.292330 12723 rebalancer_tool.cc:1136] tablet cf1f28d15c094fd98fac1b91d2785744: '861f629e2d134ff08f25fa553dfb5bda' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move completed: OK
I20240501 08:12:48.360086 12723 rebalancer_tool.cc:1136] tablet 8f27eb42e75247c2aa142d3028e6ca2a: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move completed: Incomplete: tablet 8f27eb42e75247c2aa142d3028e6ca2a, TS c3747ec2a93c4684962f215616e896ed -> TS bb2f0099fb794400b5c24927e213e5d8 move failed, target replica disappeared
I20240501 08:12:48.458364 12723 rebalancer_tool.cc:1136] tablet af24b8ab4d0c43ef986d89d4aefc263c: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '' move completed: OK
I20240501 08:12:48.678133 12723 rebalancer_tool.cc:1136] tablet b72323b8b99f4fd1aedcce69089f6204: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '441e281fafbf42f8938c3f2f42ff7527' move completed: OK
I20240501 08:12:48.744167 12723 rebalancer_tool.cc:1136] tablet 6624441a1f1143648913595f905f8039: '06c1439c9ab8415485eff20566811ede' -> 'bb2f0099fb794400b5c24927e213e5d8' move completed: OK
I20240501 08:12:48.831368 12723 rebalancer_tool.cc:1136] tablet ae946c165cd34853917e5c8a244b6408: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move completed: OK
I20240501 08:12:48.863598 12723 rebalancer_tool.cc:1136] tablet 850d4e1020e243f4b6fa113d2946846d: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move completed: OK
I20240501 08:12:49.080658 12723 rebalancer_tool.cc:1084] tablet 03d0f89e1d8f40fe84d275b117d8145f: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move ignored: Illegal state: tablet 03d0f89e1d8f40fe84d275b117d8145f: replica bb2f0099fb794400b5c24927e213e5d8 is already present
I20240501 08:12:49.080861 12723 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): unacceptable health status UNAVAILABLE

I20240501 08:12:49.767032 10619 raft_consensus.cc:1261] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Refusing update from remote peer c9f609481ded4d1aad9dd5f2341fd931: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:12:49.767686 15264 consensus_queue.cc:1035] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:12:49.768723 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:49.851089 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:49.875252 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:49.972443 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:49.978772 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:49.984994 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:49.990890 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.033447 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:50.086620 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.107131 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.198249 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:50.236356 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.273701 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.329496 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.362342 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.417330 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.445240 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:50.448871 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:50.461791 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.498173 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:50.552191 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.553689 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:50.594383 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.881939 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:50.906172 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:50.952334 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:50.957216 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:51.017906 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:51.367273 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:51.410712 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:51.428717 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:51.470171 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:51.552794 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:51.619221 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:51.773844 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:51.856714 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:51.889276 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:51.890340 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:51.895699 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:51.919085 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:51.938720 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:51.976754 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:12:52.000881 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.150668 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:12:52.151748 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.236174 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:12:52.237295 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:52.298010 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:52.344763 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:12:52.345806 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:52.356911 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:12:52.357772 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.360415 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.417021 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:52.442091 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.478127 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:52.512372 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.530596 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.579284 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.653242 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.658272 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.765733 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:52.803848 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.824771 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:52.848009 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:52.882387 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:52.950726 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:53.000831 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:53.041319 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:53.066599 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:53.225244 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:53.226863 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:53.250270 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:53.316092 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:53.615110 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:53.679195 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:53.693608 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:53.770486 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:53.804277 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:53.821552 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:54.081025 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:54.095366 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:54.179864 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:54.210937 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:54.250110 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:54.296012 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:54.326452 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:54.350181 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:12:54.351212 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:54.401921 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:54.410683 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:12:54.411692 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:12:54.538863 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:54.590550 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:54.695133 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:54.719112 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:54.768947 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:54.817267 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:54.962348 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:55.018630 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.027364 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.033795 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:55.035203 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.055074 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.156270 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.259093 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.296280 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.305032 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:55.355669 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:55.361974 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:55.373327 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.376992 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.395067 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.494771 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:55.588768 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.612301 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.780721 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:55.863622 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:55.910940 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:55.915261 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:55.959581 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:56.058950 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:56.165102 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:56.354363 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:56.416688 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 08:12:56.418249 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:56.441596 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:56.450445 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:56.454272 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:56.525161 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:56.605935 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:56.736179 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:56.846808 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:56.869705 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:56.902832 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:56.970676 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:57.003507 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:57.004912 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:57.011487 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:12:57.071558 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.273528 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:12:57.299019 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:12:57.299875 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:57.321169 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:57.341476 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.368336 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:12:57.369473 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.377568 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:57.400064 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.420471 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.466892 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.574527 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:12:57.604718 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.609730 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.616818 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:57.756839 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.784565 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:57.805473 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.809299 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.810456 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.844409 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:57.923269 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:57.990646 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:58.025763 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:58.198977 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:58.212975 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:58.270301 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:58.461512 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:58.483737 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:58.576637 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:58.679121 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:58.696861 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:58.742483 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:58.889420 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:58.959513 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:59.042268 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:12:59.065021 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:59.134471 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:59.183415 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:59.193096 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:59.251222 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:59.469249 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:12:59.507154 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:59.659054 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:59.689554 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:12:59.690680 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:59.739408 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:12:59.757140 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:12:59.766106 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:12:59.767256 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:12:59.774009 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:12:59.785818 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:12:59.793118 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:12:59.853870 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.056185 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:00.091349 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.167266 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.208015 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:00.220676 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.228425 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.274255 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.300233 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.308549 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:00.375550 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:00.417976 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.489051 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.496170 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:00.631876 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:00.770889 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:00.795625 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:00.822989 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:00.824661 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:01.102835 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:01.198082 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:01.209204 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:01.233263 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:01.241462 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:01.391369 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:01.406302 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:01.562116 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:01.584136 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:01.726675 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:01.728695 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:01.762428 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:01.785470 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:01.918722 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:01.939126 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.064742 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:02.104094 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.205526 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:02.227733 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:02.239640 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.271381 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:02.284623 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:13:02.326493 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.326731 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:13:02.343799 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:13:02.407441 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:13:02.461143 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.462008 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:02.477021 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:02.549561 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.654038 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:13:02.720995 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.745890 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.792367 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.821444 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:02.826179 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:02.880924 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:02.912931 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:03.011798 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.012030 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:03.074970 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.146097 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.179116 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.379833 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.414357 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:03.440888 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:03.454506 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:03.599606 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:03.660638 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.677919 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.781071 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:03.891397 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:03.974527 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:04.005901 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.021045 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:04.021091 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.213835 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:04.423316 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.451898 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:04.453359 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:04.463243 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:04.532618 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.553246 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:04.624261 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.625268 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.644008 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:04.793040 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:04.869915 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:04.930161 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:04.950850 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.965750 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:04.995631 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:04.997124 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:05.065227 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:05.072693 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.074129 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:05.075145 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:13:05.115691 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.166268 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.237687 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.240455 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.324960 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:05.387841 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:05.487812 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:05.539170 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.595598 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.608904 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.637012 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:05.669898 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.714861 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.766777 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:05.821521 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:05.914230 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:05.926364 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:06.067224 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:06.184448 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:06.229947 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:06.232648 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:06.323227 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:06.351841 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:06.372678 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:06.585404 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:06.586936 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:06.598868 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:06.707772 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:06.746666 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:06.838349 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:06.891467 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:06.959146 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:07.016305 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:07.055214 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.086969 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.159595 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:07.303965 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:07.309230 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:13:07.322818 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:07.346238 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:07.361605 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.477030 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:07.488271 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:13:07.502095 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:13:07.552438 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.683094 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:07.692822 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:07.704196 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:13:07.717275 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:13:07.729262 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.857558 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.860513 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.924013 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.965157 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:07.968598 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:07.994076 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.036139 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.083364 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:08.150738 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:08.213268 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.251258 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.301024 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:08.312930 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.400872 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:08.473699 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.518333 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:08.562152 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.573141 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:08.602010 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:08.849937 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:08.850178 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:08.935340 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:09.011505 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:09.140779 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:09.222235 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:09.286020 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:09.310997 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:09.327498 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:09.337335 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:09.389359 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:09.530285 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:09.564930 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:09.635689 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:09.730688 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:09.760900 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:09.780510 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:09.893817 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:09.996814 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.143908 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:10.166702 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:10.192106 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:10.259364 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:10.283632 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.288661 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.294982 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:10.315315 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.328393 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.379222 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:10.380429 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:13:10.497493 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.591436 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:10.634804 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.664242 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.677189 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.732722 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.745616 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:10.802511 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:10.827795 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.857926 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:10.913578 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:11.036708 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:11.087399 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:11.182173 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:11.322459 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:11.400327 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:11.507181 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:11.551805 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:11.589603 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:11.687482 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:11.768651 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:11.781152 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:11.786202 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:11.809504 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:11.910261 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:11.969846 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:11.972103 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:12.139909 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:12.165230 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:12.187219 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:12.323730 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:13:12.348695 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:12.365890 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:12.435616 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:12.508734 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:13:12.555168 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 08:13:12.555944 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:12.699487 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:12.704005 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:12.716594 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:12.739524 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:13:12.790544 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:12.809614 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:12.809654 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:12.825441 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:12.882548 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:12.923693 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:13:12.951195 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.005884 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:13.125821 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:13.319442 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:13.324446 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:13.340313 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.345916 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.377197 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.459228 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.469360 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.493873 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.509147 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:13.623091 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:13.671399 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:13.702765 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:13.883388 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:13.886910 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:14.077222 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:14.094441 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:14.147306 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:14.250481 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:14.310902 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:14.328500 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:14.338187 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:14.342013 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:14.459272 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:14.533428 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:14.675652 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:14.693922 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:14.736706 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:14.743736 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:14.748143 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:14.758533 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:14.806463 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.044543 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.123135 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.159111 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:15.265851 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:15.272827 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:15.320271 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:15.320631 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:13:15.478222 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:15.546173 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.583846 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:15.593336 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.599730 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.677444 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:15.691951 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:15.722818 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:15.732494 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.764217 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.828406 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:15.902133 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:16.052618 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:16.068991 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:16.084228 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:16.098471 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:16.123643 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:16.182341 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:16.247349 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:16.260447 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:16.336349 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:16.545547 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:16.705134 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:16.736465 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:16.760614 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:16.764799 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:16.793172 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:16.872882 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:16.890033 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:16.909327 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:16.981770 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:17.144225 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:17.227430 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:17.227687 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:17.279531 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:17.323127 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:17.382987 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:13:17.391409 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:17.544895 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:17.547297 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:17.585405 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:13:17.587584 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:13:17.588490 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:17.592804 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:17.606577 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:17.620437 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:17.761034 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:13:17.778101 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:13:17.854771 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:18.082698 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:18.094264 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.109019 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.125339 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.135176 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.155522 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:18.157413 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:18.353396 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.355136 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.373034 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:18.390784 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.590759 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.599859 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:18.635735 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:18.651801 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:18.717573 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:18.773211 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:18.953253 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:18.965483 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:19.084161 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:19.220994 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:19.221140 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:19.238577 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:19.244767 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:19.334693 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:19.374995 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:19.458374 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:19.560667 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:19.573052 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:19.636255 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:19.688803 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:19.698000 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:19.782609 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:19.866703 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.001533 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.007190 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:20.040694 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.078164 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:20.253799 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:20.312253 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:20.312758 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.338956 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:13:20.425415 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:20.599092 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:20.658110 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.672987 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.687983 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.692725 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:20.704299 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.768990 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.797437 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:20.823782 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:20.848891 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:20.993636 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:21.029793 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:21.038729 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:21.040079 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:21.064671 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:21.092366 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:21.139274 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:21.162663 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:21.236533 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:21.263469 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:21.290244 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:21.392069 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:21.449918 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:21.657649 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:21.687462 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:21.712810 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:21.747357 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:21.787184 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:21.846472 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:21.889770 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:21.964010 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:21.979905 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:22.024844 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:22.109011 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:22.117174 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:22.135684 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:22.278417 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:22.348641 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:22.389832 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:22.393405 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:13:22.448484 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:22.495272 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:22.623420 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 08:13:22.652724 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:22.662444 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:13:22.672567 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:22.728621 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:22.766610 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:13:22.813180 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:22.823000 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:13:23.012413 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:23.080037 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:23.097278 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.150542 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.175781 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.182129 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:23.238502 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.310588 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.320375 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:23.343477 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.352828 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.353286 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.432472 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.517264 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:23.563153 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:23.582402 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:23.677328 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:23.698812 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:23.906450 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:23.983817 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:24.068372 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:24.073630 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:24.127157 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:24.180454 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:24.180642 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:24.276852 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:24.278247 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:24.352553 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:24.456693 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:24.468066 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:24.522567 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:24.543324 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:24.594184 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:24.663334 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:24.796590 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:24.879827 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:24.921650 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:25.055395 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.077916 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.232384 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.253423 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:25.284595 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:25.341668 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:25.456812 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:25.545033 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.553666 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:13:25.570372 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.571841 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.743224 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.756376 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.759341 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:25.787977 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:25.911351 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:25.939740 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:25.946266 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:25.987023 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:26.004703 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:26.009421 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:26.079525 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:26.256036 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:26.265635 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:26.288319 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:26.369071 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:26.381636 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:26.401366 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:26.517078 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:26.614230 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:26.635048 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:26.647233 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:26.713508 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:26.842070 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:26.874441 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:26.875996 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:26.883699 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:26.926064 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:27.133189 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:27.137656 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:27.206537 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:27.325450 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:27.371160 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:27.375591 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:27.457942 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:13:27.514652 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:27.639617 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:27.671212 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:13:27.736572 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:13:27.762692 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:27.766003 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:27.835574 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:27.872217 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:13:27.873448 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:27.945964 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:27.950062 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:27.954437 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:13:27.998544 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.063258 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:28.074090 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.225927 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.267786 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:28.339546 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:28.374817 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:28.382573 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.457366 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:28.537634 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.545363 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:28.552145 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.578253 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.600836 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.668004 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.766214 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:28.798185 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.828513 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:28.921765 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:29.023167 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:29.091478 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:29.203094 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:29.247402 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:29.258931 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:29.327667 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:29.331370 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:29.395046 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:29.506767 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:29.513859 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:29.571179 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:29.663167 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:29.730752 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:29.870074 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:30.060292 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:30.092345 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:30.096800 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.161679 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:13:30.162673 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.241907 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:30.266705 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.290730 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:30.502758 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:30.663616 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:30.666669 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:30.672542 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.693150 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:30.761018 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.794757 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.818779 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.840500 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:30.916813 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:30.945618 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.078027 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:31.094296 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.135013 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.143033 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.163821 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:31.193181 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:31.207794 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:31.208599 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.209650 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.410754 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.456514 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:31.544476 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:31.565730 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:31.578883 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:31.637171 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:31.658937 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:31.723217 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:31.724100 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.746608 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:31.754956 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:31.860769 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:31.866523 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:31.984922 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:32.021808 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:32.108305 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:32.169621 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:32.423985 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:32.520483 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:32.575870 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:32.624101 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:32.634662 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:13:32.653508 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:13:32.654441 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:32.668351 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:32.737644 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:13:32.741492 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:13:32.883076 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:13:32.919366 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:32.981503 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:33.125936 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:33.153626 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:33.180145 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.183149 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.196182 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:33.200007 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:33.233749 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.243651 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.251211 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.282202 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.365124 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.367141 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.560931 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:33.582325 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:33.617218 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.715004 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:33.749594 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:33.749773 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:33.900877 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:33.988626 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:34.028026 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:34.103384 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:34.142736 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:34.144104 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:34.180914 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:34.265022 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:34.294365 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:34.352502 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:34.370656 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:34.418535 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:34.470783 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:34.561015 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:34.715322 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:34.792356 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:34.895746 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:34.990906 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:35.016947 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:13:35.174121 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.215111 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.318718 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:35.339277 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:35.469822 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:35.485082 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.535946 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.545640 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:35.556701 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:35.706575 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.750005 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:35.779462 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:35.832681 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.853245 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.885672 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.927165 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:35.954201 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:36.014314 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:36.048797 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:36.076579 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:36.127099 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:36.133273 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:36.188402 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:36.296123 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:36.370406 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:36.395206 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:36.409397 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:36.477133 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:36.596942 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:36.620507 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:36.724030 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:36.817809 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:36.818647 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:36.836174 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:36.923875 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:36.934326 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:37.043776 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:37.068456 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:37.170653 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:37.195590 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:37.212100 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:37.236977 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:37.446074 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:37.478679 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:13:37.609330 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:37.736915 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:37.780438 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:13:37.787266 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:37.840801 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:13:37.841642 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:37.900655 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:13:37.945866 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:13:37.946671 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:37.974157 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:38.010082 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.036742 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.058238 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.066489 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:38.102041 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.158239 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:38.225543 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:38.370365 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.388975 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.445914 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.489235 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.528699 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.535531 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:38.581897 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:38.586233 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:38.612816 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:38.651777 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.752524 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:38.794481 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.894744 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.931631 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:38.984186 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:38.992139 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:39.112295 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:39.176367 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:39.185725 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:39.259127 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:39.259706 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:39.524189 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:39.537581 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:39.549252 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:39.573094 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:39.709142 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:39.758801 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:39.805613 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:39.988279 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:40.021420 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:40.149585 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:40.243083 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:40.321205 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:13:40.410634 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:40.452572 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:40.482692 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:40.499192 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:40.512229 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:40.538681 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:40.603547 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:40.678951 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:40.754318 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:40.794315 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:40.845341 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:40.850919 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:40.902873 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:40.978807 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:41.010196 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.044793 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:41.073055 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:41.075081 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.096596 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.144178 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.261183 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.323580 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.334631 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.395362 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:41.432945 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:41.445039 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:41.632234 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:41.675565 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:41.717972 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.866852 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:41.949102 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:41.950131 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:41.965889 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:41.977129 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:42.273885 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:42.285212 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:42.295867 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:42.302037 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:42.302065 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:42.458914 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:42.518368 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:42.519426 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:42.744537 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:42.748721 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:42.758359 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:13:42.850123 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:13:42.887293 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:13:42.911689 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:13:42.912804 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:42.915822 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:42.997462 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:13:42.998276 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:43.035008 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:43.091094 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.142397 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.181530 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:43.240377 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.263331 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.296386 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.367599 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:43.388171 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.391361 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:43.391542 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:43.423578 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.554692 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.612567 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:43.616753 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.680644 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.697147 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.706686 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:43.865677 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:43.925623 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:43.931838 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:43.942935 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:43.949165 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:44.012465 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:44.118636 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:44.201424 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:44.317435 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:44.456179 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:44.539672 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:44.684695 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:44.743192 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:44.763204 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:44.777361 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:44.803982 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:44.953420 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:45.051100 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:45.071926 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:45.179400 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:45.190451 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:13:45.296590 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:45.489115 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:45.542969 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:45.590503 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:45.650334 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:45.700323 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:45.707573 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:45.728703 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:45.739742 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:45.772871 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:45.820247 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:45.897889 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:45.913388 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:45.930194 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:45.982662 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.003441 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.019289 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.046679 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.117193 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:46.207592 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.232147 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.254696 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.277093 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.280334 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:46.290501 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:46.364768 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:46.398713 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:46.447397 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:46.660882 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:46.778571 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:46.836879 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:46.960681 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:47.108502 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:47.169023 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:47.170171 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:47.245167 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:47.320360 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:13:47.321267 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:13:47.333329 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:47.350266 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:47.355006 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:47.389797 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:47.683370 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:47.820044 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:47.877754 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:47.905223 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:47.916852 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:13:47.924991 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:13:47.936892 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:47.955551 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:48.153291 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.180464 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.180750 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:13:48.200264 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:13:48.201038 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:48.238937 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.254832 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.281078 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:48.287521 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:48.297641 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:48.301505 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:48.306993 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.338973 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.441421 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.461756 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.518926 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.545866 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.556917 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.557695 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:48.579432 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.763453 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:48.795447 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:48.820600 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:48.858109 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:48.892747 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:48.949713 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:49.096724 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:49.192772 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:49.296339 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:49.301124 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:49.698310 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:49.707033 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:49.721752 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:49.798099 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:49.856528 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:49.860991 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:49.896319 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:50.047197 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:13:50.180333 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.205447 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:50.268222 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:50.310029 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:50.349192 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:50.366822 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.490444 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.503265 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:50.685981 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:50.707466 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:50.735594 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.835152 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:50.850570 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.878023 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.882951 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:50.923367 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:50.933296 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.938771 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:50.999665 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.027057 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.061786 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.117164 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.146750 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.233039 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.261106 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:51.298818 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.365273 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:51.376998 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:51.401566 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:51.479265 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:51.530673 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:51.861028 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:51.917788 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:51.967136 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:52.075505 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:52.167505 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:52.305169 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:52.364501 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:52.414597 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:52.417213 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:52.423439 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:52.425617 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:52.472975 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:52.474205 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:13:52.561384 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:52.634454 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:52.663897 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:52.772300 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:52.836686 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:52.934698 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:13:52.953847 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:53.047415 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.073191 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:53.121526 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:53.180402 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:53.205319 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:13:53.270987 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:13:53.271973 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.280253 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:53.316239 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.344151 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.399408 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:13:53.400190 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:53.422900 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:53.445804 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.496939 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.507824 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.634410 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:53.694214 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.724850 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:53.730597 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.733743 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.761948 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.814347 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.850255 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:53.851540 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:53.861896 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:53.881582 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:53.935384 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:54.194841 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:54.248265 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:54.300402 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:54.312345 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:54.313516 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:54.668462 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:54.729029 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:54.750303 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:54.760540 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:54.789067 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:54.897699 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:54.922355 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.010010 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:13:55.011210 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.079267 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.152686 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:55.197021 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:55.228832 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:55.237541 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:55.291585 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.613839 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:55.637646 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:55.699211 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:55.703863 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:55.736614 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.767051 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:55.820878 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.835744 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:55.869590 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.883903 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.924885 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.970593 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:55.988775 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:56.025661 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:56.087635 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:56.095628 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:56.220628 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:56.227448 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:56.235869 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:56.352224 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:56.378908 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:56.492641 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:56.504101 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:56.508057 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:56.624769 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:56.759268 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:56.844410 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:56.844928 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:57.019984 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:57.192556 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:57.282608 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:57.314502 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:57.332726 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:57.438575 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:57.478816 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:57.553241 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:57.554270 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:57.557930 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:57.580135 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:13:57.581226 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:13:57.607019 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:57.621732 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:57.772923 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:57.857581 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:57.895258 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:13:57.966015 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:13:57.983130 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:58.083652 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:58.095479 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.102936 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:58.214766 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:13:58.258442 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.268841 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:58.336411 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:13:58.350693 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.400877 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.403806 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:13:58.404594 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:58.417938 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:13:58.470897 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.502856 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:58.521684 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:58.574231 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.647374 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.650979 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:58.672221 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.673736 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.766081 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.821331 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:58.850145 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:58.959703 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:58.962065 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:59.075559 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:59.160071 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:59.186550 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:59.264585 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:13:59.368250 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:59.437619 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:59.443684 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:59.649209 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:13:59.716876 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:59.820358 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:13:59.876724 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:13:59.937212 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:13:59.971760 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:13:59.982132 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:00.089957 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:14:00.109440 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:14:00.123140 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:00.195385 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:00.286919 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:00.353740 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:00.430559 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:00.490528 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:00.542454 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:00.621100 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:00.653966 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:00.663313 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:00.717716 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:00.786854 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:00.794135 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:00.904385 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:00.976001 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:00.998980 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:01.038599 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.051285 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.053133 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.162946 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:01.203603 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.260320 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:01.286648 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.296938 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.436041 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.458108 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:01.467542 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.486166 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:01.582202 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:01.724992 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:01.749958 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.753942 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:01.911541 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:01.994956 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:02.017016 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:02.104890 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:02.179683 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:02.342900 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:02.346312 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:02.452172 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:02.510473 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:02.611071 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:02.654143 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:02.665158 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:02.668954 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:02.670029 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:14:02.735239 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:02.757359 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:02.916188 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:02.963847 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:02.991856 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.008644 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:03.009488 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.092831 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:03.124822 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:03.240746 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.241565 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:03.252663 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:03.253536 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.285919 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.288024 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:03.314642 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.360190 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:14:03.409977 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:03.534463 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.541663 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:03.568445 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:14:03.619707 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:03.627205 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.637928 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:03.726912 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.775202 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.880486 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.881829 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:03.967901 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:04.031912 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:04.034543 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:04.075325 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:04.092360 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:04.194460 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:04.292239 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:04.499665 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:04.530222 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:04.564780 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:04.684069 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:04.801846 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:04.930028 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:04.950335 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:04.952647 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:04.953337 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:05.069828 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:05.135651 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.162122 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.185950 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:05.223963 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:14:05.300637 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:05.359669 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:05.446794 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.448067 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.455825 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.554592 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:05.565044 10210 maintenance_manager.cc:419] P ee59fa3365af4b91907d4ab87092780e: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033375
I20240501 08:14:05.595253 10209 maintenance_manager.cc:643] P ee59fa3365af4b91907d4ab87092780e: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.030s	user 0.022s	sys 0.004s Metrics: {"bytes_written":21148,"cfile_init":1,"dirs.queue_time_us":412,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":794,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":753,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":481,"threads_started":2,"wal-append.queue_time_us":303}
I20240501 08:14:05.596408 10210 maintenance_manager.cc:419] P ee59fa3365af4b91907d4ab87092780e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10424 bytes on disk
I20240501 08:14:05.597064 10209 maintenance_manager.cc:643] P ee59fa3365af4b91907d4ab87092780e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240501 08:14:05.620891 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.634986 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:05.664330 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:05.695948 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:05.751871 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.831599 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:05.875473 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.883008 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.922348 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:05.994925 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:06.049142 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:06.112422 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:06.159348 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:06.162904 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:06.199452 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:06.378185 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:06.430321 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:06.552307 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:06.606320 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:06.615541 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:06.630125 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:06.757277 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:06.803609 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:06.866652 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:06.976276 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:06.989707 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:07.087281 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:07.116346 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:07.163028 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:07.335191 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:07.420809 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:07.449648 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:07.511986 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:07.583642 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:07.663246 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:07.734592 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:07.735342 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:07.740633 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:07.761020 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:07.767668 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:14:07.885699 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:07.924845 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:08.053210 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.063176 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.071442 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:14:08.072423 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:08.165544 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.170078 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.173401 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:08.220599 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:08.331737 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:08.348477 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:08.355825 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.428757 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:08.453622 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.455127 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:14:08.455896 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.466812 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:08.547786 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.593152 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.610544 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:08.670399 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:08.699056 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 08:14:08.755209 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:08.790549 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:08.834780 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:08.994455 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:09.173148 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:09.215602 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:09.228483 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:09.268517 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:09.296856 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:09.337059 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:09.404007 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:09.461277 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:09.463145 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:09.635393 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:09.760036 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:09.832499 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:09.835848 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:09.845592 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.081584 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:10.163060 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:10.188728 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.196940 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:10.299062 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:10.360231 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:10.393731 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:10.397084 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.557286 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.589663 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:14:10.643280 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:10.661829 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.743568 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.745383 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.753468 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:10.845091 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.881629 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:10.906488 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:10.968390 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:10.985111 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:10.987462 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.006327 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.024534 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.025691 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.034070 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:11.256856 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:11.372951 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:11.456878 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:11.551738 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:11.761610 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.776664 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.814194 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.822342 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:11.861557 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:11.864120 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:11.914016 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:11.941624 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:12.070467 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:12.084306 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:12.191646 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:12.266067 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:12.319208 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:12.525008 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:12.548137 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:12.557237 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:12.634220 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:12.662024 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:12.690156 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:12.701865 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:12.896101 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:12.925347 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:12.983265 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:13.094409 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:13.151014 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:13.152045 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.219856 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:13.227133 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.266778 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:13.269884 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:14:13.270768 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.298096 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:14:13.342015 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(84789f1228f64be5b6976a9d52f64bd0): perf score=0.033341
I20240501 08:14:13.344878 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(84789f1228f64be5b6976a9d52f64bd0): perf score=0.033341
W20240501 08:14:13.348332 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.399345 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:13.400242 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.409083 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:13.416354 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:13.430302 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(84789f1228f64be5b6976a9d52f64bd0) complete. Timing: real 0.088s	user 0.064s	sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":308,"dirs.queue_time_us":6998,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":7352,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17620,"threads_started":3,"wal-append.queue_time_us":10346}
I20240501 08:14:13.432453 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0): 4581 bytes on disk
I20240501 08:14:13.434005 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 08:14:13.450870 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(84789f1228f64be5b6976a9d52f64bd0) complete. Timing: real 0.105s	user 0.063s	sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":1960,"dirs.queue_time_us":6861,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":7810,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3663,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19977,"threads_started":3,"wal-append.queue_time_us":10831}
I20240501 08:14:13.453372 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0): 4581 bytes on disk
I20240501 08:14:13.454304 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240501 08:14:13.479138 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(84789f1228f64be5b6976a9d52f64bd0): perf score=0.033382
W20240501 08:14:13.481596 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:14:13.484457 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:14:13.522681 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(50b7150bbeb645d1b00d1a561c3276d1): perf score=0.033395
W20240501 08:14:13.525876 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.572492 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.583109 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:13.676100 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:13.682798 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:13.722220 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(84789f1228f64be5b6976a9d52f64bd0) complete. Timing: real 0.242s	user 0.052s	sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":4146,"dirs.queue_time_us":5373,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":910,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14128,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":20126,"threads_started":3,"wal-append.queue_time_us":10961}
I20240501 08:14:13.726426 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0): 4581 bytes on disk
I20240501 08:14:13.729980 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240501 08:14:13.743007 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:13.817209 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:14:13.838445 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:13.913712 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(50b7150bbeb645d1b00d1a561c3276d1) complete. Timing: real 0.390s	user 0.074s	sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":18173,"dirs.queue_time_us":6254,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1024,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":903,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":95802,"threads_started":3,"wal-append.queue_time_us":64285}
W20240501 08:14:13.917318 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:13.924156 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(af24b8ab4d0c43ef986d89d4aefc263c): perf score=0.033504
W20240501 08:14:14.028978 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:14.079788 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(af24b8ab4d0c43ef986d89d4aefc263c) complete. Timing: real 0.155s	user 0.046s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1375,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":20167,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":22691,"threads_started":1,"wal-append.queue_time_us":7713}
I20240501 08:14:14.081136 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(af24b8ab4d0c43ef986d89d4aefc263c): 4921 bytes on disk
I20240501 08:14:14.088860 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(af24b8ab4d0c43ef986d89d4aefc263c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 08:14:14.090130 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(50b7150bbeb645d1b00d1a561c3276d1): 4834 bytes on disk
I20240501 08:14:14.092119 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(50b7150bbeb645d1b00d1a561c3276d1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240501 08:14:14.127921 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:14.227411 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:14.293996 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.132:40093, user_credentials={real_user=slave}} blocked reactor thread for 51680.1us
W20240501 08:14:14.331756 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:14.338850 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:14.350267 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:14.435680 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:14.438381 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(c665d7d20360440a9103e245f5f0900c): perf score=0.033359
I20240501 08:14:14.459633 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(850d4e1020e243f4b6fa113d2946846d): perf score=0.033359
W20240501 08:14:14.469704 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:14.473721 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:14:14.485425 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(4b924c6344714a92ac37ae28961b02f5): perf score=0.033367
I20240501 08:14:14.515687 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(c665d7d20360440a9103e245f5f0900c) complete. Timing: real 0.077s	user 0.036s	sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12839,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1503,"drs_written":1,"lbm_read_time_us":344,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1249,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21178,"threads_started":2,"wal-append.queue_time_us":6294}
I20240501 08:14:14.517117 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c): 4921 bytes on disk
I20240501 08:14:14.517705 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240501 08:14:14.563321 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(850d4e1020e243f4b6fa113d2946846d) complete. Timing: real 0.103s	user 0.044s	sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":72058,"compiler_manager_pool.run_cpu_time_us":28,"compiler_manager_pool.run_wall_time_us":27,"dirs.queue_time_us":6080,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1224,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":833,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":8576,"thread_start_us":17261,"threads_started":2,"wal-append.queue_time_us":11364}
I20240501 08:14:14.564797 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(850d4e1020e243f4b6fa113d2946846d): 4834 bytes on disk
I20240501 08:14:14.565397 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(850d4e1020e243f4b6fa113d2946846d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 08:14:14.575817 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(4b924c6344714a92ac37ae28961b02f5) complete. Timing: real 0.090s	user 0.053s	sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":70895,"compiler_manager_pool.run_cpu_time_us":44,"compiler_manager_pool.run_wall_time_us":44,"dirs.queue_time_us":25694,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1541,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":886,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25969,"threads_started":2,"wal-append.queue_time_us":285}
I20240501 08:14:14.577020 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(4b924c6344714a92ac37ae28961b02f5): 4581 bytes on disk
I20240501 08:14:14.577644 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(4b924c6344714a92ac37ae28961b02f5) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:14:14.594837 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(4b924c6344714a92ac37ae28961b02f5): perf score=0.033400
I20240501 08:14:14.622862 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(4b924c6344714a92ac37ae28961b02f5) complete. Timing: real 0.028s	user 0.022s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":273,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1213,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":378,"threads_started":2,"wal-append.queue_time_us":260}
I20240501 08:14:14.623857 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(4b924c6344714a92ac37ae28961b02f5): 4581 bytes on disk
I20240501 08:14:14.624337 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(4b924c6344714a92ac37ae28961b02f5) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 08:14:14.625072 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(c665d7d20360440a9103e245f5f0900c): perf score=0.033406
W20240501 08:14:14.633855 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:14.639566 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:14:14.658627 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(c665d7d20360440a9103e245f5f0900c) complete. Timing: real 0.033s	user 0.027s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1298,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":233,"threads_started":1,"wal-append.queue_time_us":326}
I20240501 08:14:14.659655 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c): 4921 bytes on disk
I20240501 08:14:14.660094 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240501 08:14:14.747323 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:14.808768 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:14.838989 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:14.916350 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:14.969141 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:14.993393 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:15.097023 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:15.281041 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:15.303398 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:15.368949 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.386934 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:15.518558 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.611862 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.630867 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.638595 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:15.697206 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.771983 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:14:15.794651 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.824934 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:15.863981 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:15.869212 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.906227 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:15.943779 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:15.983201 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:16.019515 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:16.111038 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:16.133399 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:16.149942 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:16.217957 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:16.293098 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:16.323843 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(e99538651cf64510a811ba01c6c8fb40): perf score=0.033337
I20240501 08:14:16.335626 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(e99538651cf64510a811ba01c6c8fb40): perf score=0.033343
I20240501 08:14:16.374482 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(e99538651cf64510a811ba01c6c8fb40) complete. Timing: real 0.050s	user 0.027s	sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":582,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1570,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":745,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4682,"threads_started":2,"wal-append.queue_time_us":3701}
I20240501 08:14:16.375764 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(e99538651cf64510a811ba01c6c8fb40): 4581 bytes on disk
I20240501 08:14:16.376353 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(e99538651cf64510a811ba01c6c8fb40) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240501 08:14:16.377295 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(92153666aebe47ddb080278cf24d1570): perf score=0.033350
I20240501 08:14:16.409868 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(e99538651cf64510a811ba01c6c8fb40) complete. Timing: real 0.074s	user 0.044s	sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6374,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6858,"threads_started":2,"wal-append.queue_time_us":341}
I20240501 08:14:16.411000 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(e99538651cf64510a811ba01c6c8fb40): 4581 bytes on disk
I20240501 08:14:16.411700 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(e99538651cf64510a811ba01c6c8fb40) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 08:14:16.416739 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(e99538651cf64510a811ba01c6c8fb40): perf score=0.033362
I20240501 08:14:16.430078 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(92153666aebe47ddb080278cf24d1570) complete. Timing: real 0.053s	user 0.038s	sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1409,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":994,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6388,"threads_started":1,"wal-append.queue_time_us":6313}
I20240501 08:14:16.431792 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(92153666aebe47ddb080278cf24d1570): 4921 bytes on disk
I20240501 08:14:16.432495 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(92153666aebe47ddb080278cf24d1570) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240501 08:14:16.456722 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:16.464910 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(e99538651cf64510a811ba01c6c8fb40) complete. Timing: real 0.048s	user 0.030s	sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":243,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1323,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":645,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":335,"threads_started":2,"wal-append.queue_time_us":241}
I20240501 08:14:16.465936 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(e99538651cf64510a811ba01c6c8fb40): 4581 bytes on disk
I20240501 08:14:16.466372 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(e99538651cf64510a811ba01c6c8fb40) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240501 08:14:16.471109 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:16.481478 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:16.524379 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(8f27eb42e75247c2aa142d3028e6ca2a): perf score=0.033391
W20240501 08:14:16.533715 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:14:16.572067 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(8f27eb42e75247c2aa142d3028e6ca2a) complete. Timing: real 0.047s	user 0.033s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":265,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1508,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":361,"threads_started":2,"wal-append.queue_time_us":264}
I20240501 08:14:16.573122 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(8f27eb42e75247c2aa142d3028e6ca2a): 4834 bytes on disk
I20240501 08:14:16.573558 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(8f27eb42e75247c2aa142d3028e6ca2a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 08:14:16.760426 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:16.873052 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:16.890918 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:16.892778 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:16.901312 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:16.907747 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:16.957056 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:16.973189 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:17.171775 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:17.275789 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:17.322289 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:17.417753 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:17.451656 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:17.454294 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:17.483211 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:17.500998 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:17.757997 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:17.840776 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:17.906978 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:17.980244 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:18.004519 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:14:18.105844 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.118868 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:18.137671 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.160200 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:18.160953 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.302932 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:18.352481 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.361522 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.401374 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:14:18.402241 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.415691 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:18.419533 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(f5a58b7166534844b0963f4fab46dfd9): perf score=0.033346
W20240501 08:14:18.425045 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:14:18.441381 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(f5a58b7166534844b0963f4fab46dfd9): perf score=0.033355
I20240501 08:14:18.454782 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(f5a58b7166534844b0963f4fab46dfd9) complete. Timing: real 0.035s	user 0.024s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":300,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1359,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":459,"threads_started":2,"wal-append.queue_time_us":360}
I20240501 08:14:18.455806 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(f5a58b7166534844b0963f4fab46dfd9): 4832 bytes on disk
I20240501 08:14:18.456333 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(f5a58b7166534844b0963f4fab46dfd9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240501 08:14:18.474355 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(4799a93d96a64a75b1984bb85fcad7e8): perf score=0.033360
W20240501 08:14:18.474644 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:18.491143 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(f5a58b7166534844b0963f4fab46dfd9) complete. Timing: real 0.049s	user 0.034s	sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":648,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":748,"threads_started":2,"wal-append.queue_time_us":301}
I20240501 08:14:18.493089 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(3453d8d883014219b1b14c88e250736b): perf score=0.033367
W20240501 08:14:18.515251 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.521461 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:18.533212 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:14:18.534797 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.535409 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:18.543716 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:14:18.545045 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(4799a93d96a64a75b1984bb85fcad7e8) complete. Timing: real 0.070s	user 0.042s	sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1286,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":12472,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":709,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28393,"threads_started":2,"wal-append.queue_time_us":27273}
I20240501 08:14:18.546387 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(4799a93d96a64a75b1984bb85fcad7e8): 4581 bytes on disk
I20240501 08:14:18.546916 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(4799a93d96a64a75b1984bb85fcad7e8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:14:18.553601 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(3453d8d883014219b1b14c88e250736b) complete. Timing: real 0.060s	user 0.046s	sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1233,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":973,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3431,"threads_started":1,"wal-append.queue_time_us":3533}
I20240501 08:14:18.555426 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(3453d8d883014219b1b14c88e250736b): 4921 bytes on disk
I20240501 08:14:18.556049 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(3453d8d883014219b1b14c88e250736b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:14:18.557318 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(f5a58b7166534844b0963f4fab46dfd9): 4832 bytes on disk
I20240501 08:14:18.557884 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(f5a58b7166534844b0963f4fab46dfd9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240501 08:14:18.579875 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(3453d8d883014219b1b14c88e250736b): perf score=0.033395
I20240501 08:14:18.617197 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(3453d8d883014219b1b14c88e250736b) complete. Timing: real 0.037s	user 0.036s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1234,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":471,"threads_started":2,"wal-append.queue_time_us":291}
I20240501 08:14:18.618125 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(3453d8d883014219b1b14c88e250736b): 4921 bytes on disk
I20240501 08:14:18.618557 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(3453d8d883014219b1b14c88e250736b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 08:14:18.619244 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(4799a93d96a64a75b1984bb85fcad7e8): perf score=0.033404
I20240501 08:14:18.647349 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(4799a93d96a64a75b1984bb85fcad7e8) complete. Timing: real 0.028s	user 0.015s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":224,"threads_started":1,"wal-append.queue_time_us":309}
I20240501 08:14:18.648311 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(4799a93d96a64a75b1984bb85fcad7e8): 4581 bytes on disk
I20240501 08:14:18.648743 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(4799a93d96a64a75b1984bb85fcad7e8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 08:14:18.698279 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:18.747754 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.752395 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:18.867949 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:18.950023 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:14:18.960955 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:18.965554 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:19.039234 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:19.070163 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:19.297715 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:19.340781 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:19.381557 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:19.392309 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:19.439842 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:19.477018 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:19.511718 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:19.552189 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(aa6139020ac7458aa8a7ac08f63e6f26): perf score=0.033342
I20240501 08:14:19.562844 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): perf score=0.033349
W20240501 08:14:19.572955 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:19.593204 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.030s	user 0.022s	sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":289,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1367,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":529,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":402,"threads_started":2,"wal-append.queue_time_us":278}
I20240501 08:14:19.594352 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): 4581 bytes on disk
I20240501 08:14:19.594805 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 08:14:19.602756 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(aa6139020ac7458aa8a7ac08f63e6f26) complete. Timing: real 0.050s	user 0.029s	sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":275,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1737,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":703,"threads_started":2,"wal-append.queue_time_us":611}
I20240501 08:14:19.604633 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(aa6139020ac7458aa8a7ac08f63e6f26): 4834 bytes on disk
I20240501 08:14:19.605501 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(aa6139020ac7458aa8a7ac08f63e6f26) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240501 08:14:19.651980 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): perf score=0.033376
I20240501 08:14:19.688968 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.037s	user 0.032s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":283,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1220,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":716,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":377,"threads_started":2,"wal-append.queue_time_us":279}
I20240501 08:14:19.689846 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): 4581 bytes on disk
I20240501 08:14:19.690277 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 08:14:19.691017 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(bfbef7af4781448bb04b107b53d58ca0): perf score=0.033382
I20240501 08:14:19.711688 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(bfbef7af4781448bb04b107b53d58ca0): perf score=0.033392
I20240501 08:14:19.723042 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(bfbef7af4781448bb04b107b53d58ca0) complete. Timing: real 0.032s	user 0.029s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1061,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":467,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":294,"threads_started":1,"wal-append.queue_time_us":398}
I20240501 08:14:19.724243 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0): 4921 bytes on disk
I20240501 08:14:19.724697 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 08:14:19.744884 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(bfbef7af4781448bb04b107b53d58ca0) complete. Timing: real 0.033s	user 0.020s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":340,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1029,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":412,"threads_started":2,"wal-append.queue_time_us":294}
I20240501 08:14:19.745788 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(aa6139020ac7458aa8a7ac08f63e6f26): perf score=0.033396
W20240501 08:14:19.763062 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:19.789703 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(aa6139020ac7458aa8a7ac08f63e6f26) complete. Timing: real 0.044s	user 0.034s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1204,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":189,"threads_started":1,"wal-append.queue_time_us":278}
I20240501 08:14:19.790952 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0): 4921 bytes on disk
I20240501 08:14:19.791415 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 08:14:19.792435 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(aa6139020ac7458aa8a7ac08f63e6f26): 4834 bytes on disk
I20240501 08:14:19.792797 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(aa6139020ac7458aa8a7ac08f63e6f26) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240501 08:14:19.815729 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:19.858762 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:19.929575 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:19.981321 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:19.983511 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:20.028057 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:20.088325 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:20.124423 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:20.266155 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:20.332604 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:20.436640 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:20.451340 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:20.482571 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:20.613036 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:20.639955 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:14:20.642153 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:20.662869 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:20.697367 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:20.728776 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(9b6f09b0b5e84329944360d511578ed0): perf score=0.033350
W20240501 08:14:20.749572 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:20.759690 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(9b6f09b0b5e84329944360d511578ed0) complete. Timing: real 0.031s	user 0.020s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":267,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1369,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":349,"threads_started":2,"wal-append.queue_time_us":241}
I20240501 08:14:20.760609 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(9b6f09b0b5e84329944360d511578ed0): 4834 bytes on disk
I20240501 08:14:20.760979 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(9b6f09b0b5e84329944360d511578ed0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 08:14:20.780287 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:20.796936 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(9b6f09b0b5e84329944360d511578ed0): perf score=0.033363
I20240501 08:14:20.827107 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(9b6f09b0b5e84329944360d511578ed0) complete. Timing: real 0.030s	user 0.027s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":252,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1339,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":330,"threads_started":2,"wal-append.queue_time_us":230}
I20240501 08:14:20.828182 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(9b6f09b0b5e84329944360d511578ed0): 4834 bytes on disk
I20240501 08:14:20.828608 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(9b6f09b0b5e84329944360d511578ed0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 08:14:20.850443 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(cd01dd912e1e4ca9a104270f3cd004b1): perf score=0.033383
I20240501 08:14:20.860754 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(8a6ed4ea6d6941d2af0f81eaf744bcea): perf score=0.033384
W20240501 08:14:20.874495 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:20.890498 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(cd01dd912e1e4ca9a104270f3cd004b1) complete. Timing: real 0.040s	user 0.032s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":377,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1131,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":519,"threads_started":2,"wal-append.queue_time_us":327}
I20240501 08:14:20.892132 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(8a6ed4ea6d6941d2af0f81eaf744bcea): perf score=0.033389
W20240501 08:14:20.896226 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:20.898310 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:20.902101 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:20.906168 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(8a6ed4ea6d6941d2af0f81eaf744bcea) complete. Timing: real 0.045s	user 0.023s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":308,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1023,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":638,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10984,"threads_started":2,"wal-append.queue_time_us":10672}
I20240501 08:14:20.907557 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(cd01dd912e1e4ca9a104270f3cd004b1): perf score=0.033394
I20240501 08:14:20.954594 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(8a6ed4ea6d6941d2af0f81eaf744bcea) complete. Timing: real 0.062s	user 0.041s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1136,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":896,"thread_start_us":6408,"threads_started":1,"wal-append.queue_time_us":6292}
I20240501 08:14:20.957006 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(8a6ed4ea6d6941d2af0f81eaf744bcea): 4921 bytes on disk
I20240501 08:14:20.957581 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(8a6ed4ea6d6941d2af0f81eaf744bcea) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 08:14:20.959139 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(cd01dd912e1e4ca9a104270f3cd004b1): 4579 bytes on disk
I20240501 08:14:20.959659 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(cd01dd912e1e4ca9a104270f3cd004b1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 08:14:20.959838 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(cd01dd912e1e4ca9a104270f3cd004b1) complete. Timing: real 0.052s	user 0.032s	sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1066,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12723,"threads_started":1,"wal-append.queue_time_us":12804}
I20240501 08:14:20.960898 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(8a6ed4ea6d6941d2af0f81eaf744bcea): 4921 bytes on disk
I20240501 08:14:20.961496 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(8a6ed4ea6d6941d2af0f81eaf744bcea) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240501 08:14:20.962889 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(cd01dd912e1e4ca9a104270f3cd004b1): 4579 bytes on disk
I20240501 08:14:20.963368 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(cd01dd912e1e4ca9a104270f3cd004b1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240501 08:14:20.990980 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:21.033125 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:21.043315 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:21.148619 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:21.151374 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:21.156939 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:21.396736 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:21.423434 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:21.452397 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:21.480535 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:21.639168 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:21.765290 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(185a62881ea84131a0c14987e4296338): perf score=0.033335
I20240501 08:14:21.795799 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(185a62881ea84131a0c14987e4296338) complete. Timing: real 0.030s	user 0.019s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":276,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1415,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":365,"threads_started":2,"wal-append.queue_time_us":246}
I20240501 08:14:21.796716 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338): 4919 bytes on disk
I20240501 08:14:21.797120 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:14:21.832741 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(6624441a1f1143648913595f905f8039): perf score=0.033360
W20240501 08:14:21.834594 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:21.862787 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(6624441a1f1143648913595f905f8039) complete. Timing: real 0.030s	user 0.023s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1239,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":459,"threads_started":2,"wal-append.queue_time_us":306}
I20240501 08:14:21.864027 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(6624441a1f1143648913595f905f8039): 4581 bytes on disk
I20240501 08:14:21.864550 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(6624441a1f1143648913595f905f8039) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240501 08:14:21.879310 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:21.964607 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(d188d415f63f448195ae90e8e3d66e0e): perf score=0.033394
I20240501 08:14:21.967729 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(185a62881ea84131a0c14987e4296338): perf score=0.033389
W20240501 08:14:21.992700 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:22.030808 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(d188d415f63f448195ae90e8e3d66e0e) complete. Timing: real 0.066s	user 0.050s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":302,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":5434,"drs_written":1,"lbm_read_time_us":249,"lbm_reads_lt_1ms":4,"lbm_write_time_us":894,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":886,"threads_started":2,"wal-append.queue_time_us":283}
I20240501 08:14:22.032414 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(d188d415f63f448195ae90e8e3d66e0e): 4834 bytes on disk
I20240501 08:14:22.033123 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(d188d415f63f448195ae90e8e3d66e0e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240501 08:14:22.036643 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(185a62881ea84131a0c14987e4296338) complete. Timing: real 0.069s	user 0.046s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":340,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":1029,"lbm_reads_lt_1ms":4,"lbm_write_time_us":675,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":592,"threads_started":2,"wal-append.queue_time_us":414}
I20240501 08:14:22.038364 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338): 4919 bytes on disk
I20240501 08:14:22.038945 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 08:14:22.049954 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.057101 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.086436 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.089852 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:22.136510 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:22.138914 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:22.198081 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:22.376773 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:22.417069 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:22.511214 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:22.516683 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:22.675019 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:22.806651 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:14:22.807896 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.813537 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.846195 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:22.859930 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:22.881263 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.941542 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.943516 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:22.988193 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:23.032815 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:14:23.037616 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(b72323b8b99f4fd1aedcce69089f6204): perf score=0.033382
I20240501 08:14:23.043305 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(b72323b8b99f4fd1aedcce69089f6204): perf score=0.033385
I20240501 08:14:23.051918 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(ea502ae00eff4f72a74b8d25f5a11ede): perf score=0.033371
W20240501 08:14:23.100502 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:23.108562 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(ea502ae00eff4f72a74b8d25f5a11ede) complete. Timing: real 0.056s	user 0.037s	sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6439,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":12348,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":678,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17749,"threads_started":2,"wal-append.queue_time_us":11323}
I20240501 08:14:23.110078 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(ea502ae00eff4f72a74b8d25f5a11ede): 4581 bytes on disk
I20240501 08:14:23.111119 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(ea502ae00eff4f72a74b8d25f5a11ede) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 08:14:23.116245 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(b72323b8b99f4fd1aedcce69089f6204) complete. Timing: real 0.073s	user 0.044s	sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9844,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1034,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1083,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16330,"threads_started":2,"wal-append.queue_time_us":6493}
I20240501 08:14:23.117666 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(b72323b8b99f4fd1aedcce69089f6204): 4835 bytes on disk
I20240501 08:14:23.118012 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(b72323b8b99f4fd1aedcce69089f6204) complete. Timing: real 0.080s	user 0.038s	sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":20069,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":7775,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":853,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":22184,"threads_started":2,"wal-append.queue_time_us":1857}
I20240501 08:14:23.118813 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(b72323b8b99f4fd1aedcce69089f6204) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:14:23.119745 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(03d0f89e1d8f40fe84d275b117d8145f): perf score=0.033403
I20240501 08:14:23.119760 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(b72323b8b99f4fd1aedcce69089f6204): 4835 bytes on disk
I20240501 08:14:23.120306 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(b72323b8b99f4fd1aedcce69089f6204) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 08:14:23.163547 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(03d0f89e1d8f40fe84d275b117d8145f) complete. Timing: real 0.044s	user 0.030s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":285,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":719,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1287,"threads_started":2,"wal-append.queue_time_us":1193}
W20240501 08:14:23.164194 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:14:23.165226 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(ea502ae00eff4f72a74b8d25f5a11ede): perf score=0.033402
W20240501 08:14:23.166085 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:23.203577 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(ea502ae00eff4f72a74b8d25f5a11ede) complete. Timing: real 0.038s	user 0.034s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1270,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":200,"threads_started":1,"wal-append.queue_time_us":313}
I20240501 08:14:23.204864 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(03d0f89e1d8f40fe84d275b117d8145f): 4921 bytes on disk
I20240501 08:14:23.205453 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(03d0f89e1d8f40fe84d275b117d8145f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240501 08:14:23.206439 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(ea502ae00eff4f72a74b8d25f5a11ede): 4581 bytes on disk
I20240501 08:14:23.206849 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(ea502ae00eff4f72a74b8d25f5a11ede) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240501 08:14:23.291117 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:23.374483 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:23.378058 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:23.400638 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:23.410765 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:14:23.486253 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:23.543946 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:23.545684 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:14:23.558295 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:23.680536 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:23.705092 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:23.725241 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:23.740186 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:23.746201 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:23.872974 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(2a63e16f11894c2886091dcd3d204106): perf score=0.033340
I20240501 08:14:23.874631 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(bbef0147276d410da3d3830e682efe6d): perf score=0.033337
W20240501 08:14:23.892303 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:23.909304 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(2a63e16f11894c2886091dcd3d204106) complete. Timing: real 0.036s	user 0.031s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":349,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1295,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":734,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":544,"threads_started":2,"wal-append.queue_time_us":379}
I20240501 08:14:23.910909 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(2a63e16f11894c2886091dcd3d204106): 4581 bytes on disk
I20240501 08:14:23.911453 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(2a63e16f11894c2886091dcd3d204106) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 08:14:23.914489 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(bbef0147276d410da3d3830e682efe6d) complete. Timing: real 0.040s	user 0.034s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":425,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1159,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":522,"threads_started":2,"wal-append.queue_time_us":316}
I20240501 08:14:23.915514 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(bbef0147276d410da3d3830e682efe6d): 4922 bytes on disk
I20240501 08:14:23.916023 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(bbef0147276d410da3d3830e682efe6d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240501 08:14:23.918349 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:23.960790 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(cf1f28d15c094fd98fac1b91d2785744): perf score=0.033365
I20240501 08:14:23.998485 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(cf1f28d15c094fd98fac1b91d2785744) complete. Timing: real 0.037s	user 0.027s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":269,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":622,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":852,"threads_started":2,"wal-append.queue_time_us":756}
I20240501 08:14:23.999814 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(cf1f28d15c094fd98fac1b91d2785744): 4834 bytes on disk
I20240501 08:14:24.012468 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(cf1f28d15c094fd98fac1b91d2785744) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:14:24.013372 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(bbef0147276d410da3d3830e682efe6d): perf score=0.033377
I20240501 08:14:24.049516 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(bbef0147276d410da3d3830e682efe6d) complete. Timing: real 0.036s	user 0.031s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1999,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":227,"threads_started":1,"wal-append.queue_time_us":337}
I20240501 08:14:24.051112 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(bbef0147276d410da3d3830e682efe6d): 4922 bytes on disk
I20240501 08:14:24.051656 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(bbef0147276d410da3d3830e682efe6d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240501 08:14:24.066629 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:14:24.098585 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:24.116286 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(2a63e16f11894c2886091dcd3d204106): perf score=0.033401
I20240501 08:14:24.170821 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(2a63e16f11894c2886091dcd3d204106) complete. Timing: real 0.054s	user 0.031s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":376,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1145,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":495,"threads_started":2,"wal-append.queue_time_us":295}
I20240501 08:14:24.172580 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(2a63e16f11894c2886091dcd3d204106): 4581 bytes on disk
I20240501 08:14:24.173218 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(2a63e16f11894c2886091dcd3d204106) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240501 08:14:24.184023 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:24.186094 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:24.274138 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:24.302438 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:24.370687 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:24.447721 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:24.474668 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:24.505620 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:24.511837 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:24.532379 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:24.623656 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:24.694447 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:24.814620 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:24.898211 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:24.917557 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:24.956826 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:25.046509 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.087045 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.168932 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:25.276044 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:25.289417 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.291973 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:25.308157 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:25.347683 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:14:25.370093 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.382534 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:25.457943 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.478327 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.545364 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:25.602370 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:25.723237 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.774744 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.781108 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:25.811023 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:25.829105 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:25.910573 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:25.988458 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:26.060642 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(21065b7dcc18456eb1b4c2376d51eec4): perf score=0.033342
I20240501 08:14:26.095270 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.034s	user 0.030s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":275,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1236,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":743,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":426,"threads_started":2,"wal-append.queue_time_us":310}
I20240501 08:14:26.096519 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling FlushMRSOp(6d4ab8e42a484cf89b42f5f53f511265): perf score=0.033347
W20240501 08:14:26.121587 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:26.131923 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:26.159885 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: FlushMRSOp(6d4ab8e42a484cf89b42f5f53f511265) complete. Timing: real 0.063s	user 0.036s	sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1384,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":237,"threads_started":1,"wal-append.queue_time_us":321}
I20240501 08:14:26.161247 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4): 4921 bytes on disk
I20240501 08:14:26.161720 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 08:14:26.162473 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(6d4ab8e42a484cf89b42f5f53f511265): 4834 bytes on disk
I20240501 08:14:26.162832 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(6d4ab8e42a484cf89b42f5f53f511265) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 08:14:26.169792 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(ae946c165cd34853917e5c8a244b6408): perf score=0.033376
I20240501 08:14:26.179453 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(6d4ab8e42a484cf89b42f5f53f511265): perf score=0.033375
I20240501 08:14:26.181687 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling FlushMRSOp(6d4ab8e42a484cf89b42f5f53f511265): perf score=0.033376
I20240501 08:14:26.204218 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(ae946c165cd34853917e5c8a244b6408) complete. Timing: real 0.034s	user 0.034s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":423,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1303,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":607,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":930,"threads_started":2,"wal-append.queue_time_us":772}
I20240501 08:14:26.205219 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(ae946c165cd34853917e5c8a244b6408): 4582 bytes on disk
I20240501 08:14:26.205885 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(ae946c165cd34853917e5c8a244b6408) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240501 08:14:26.207077 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling FlushMRSOp(21065b7dcc18456eb1b4c2376d51eec4): perf score=0.033384
W20240501 08:14:26.207203 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:14:26.229104 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(6d4ab8e42a484cf89b42f5f53f511265) complete. Timing: real 0.049s	user 0.038s	sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1372,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":910,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":448,"threads_started":2,"wal-append.queue_time_us":315}
I20240501 08:14:26.230798 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(ae946c165cd34853917e5c8a244b6408): perf score=0.033385
I20240501 08:14:26.239529 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: FlushMRSOp(6d4ab8e42a484cf89b42f5f53f511265) complete. Timing: real 0.055s	user 0.044s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":455,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1156,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":925,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":564,"threads_started":2,"wal-append.queue_time_us":329}
I20240501 08:14:26.240895 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(6d4ab8e42a484cf89b42f5f53f511265): 4834 bytes on disk
I20240501 08:14:26.243598 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: FlushMRSOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.036s	user 0.034s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":1080,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1526,"threads_started":1,"wal-append.queue_time_us":1615}
I20240501 08:14:26.245002 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4): 4921 bytes on disk
I20240501 08:14:26.245043 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(6d4ab8e42a484cf89b42f5f53f511265) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:14:26.245606 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:14:26.273749 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(ae946c165cd34853917e5c8a244b6408) complete. Timing: real 0.043s	user 0.036s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1111,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":763,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":2816,"thread_start_us":240,"threads_started":1,"wal-append.queue_time_us":340}
I20240501 08:14:26.274847 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(6d4ab8e42a484cf89b42f5f53f511265): 4834 bytes on disk
I20240501 08:14:26.275331 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(6d4ab8e42a484cf89b42f5f53f511265) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 08:14:26.276211 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(ae946c165cd34853917e5c8a244b6408): 4582 bytes on disk
I20240501 08:14:26.276610 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(ae946c165cd34853917e5c8a244b6408) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240501 08:14:26.298498 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:26.314435 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:26.401243 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:26.468561 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:26.722784 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:26.736011 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:26.760700 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:26.827862 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:26.941915 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:26.989763 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:27.029268 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:27.086949 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:27.168892 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:27.268512 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:27.279767 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:27.326344 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:27.380697 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:27.449764 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:27.469646 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:27.473693 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:27.599608 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:27.659821 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:14:27.662775 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:27.742080 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:27.783898 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:27.815411 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:27.931905 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:27.981796 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:27.990861 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:28.043077 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:28.050065 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:28.079819 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:28.154753 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:28.197471 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:14:28.240944 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:28.259852 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:28.332998 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:28.389811 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:28.433176 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 08:14:28.461771 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:28.466398 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:28.525527 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:28.615765 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:28.635807 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:14:28.648555 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:28.704061 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:28.888391 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:28.978538 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:28.988047 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:29.045354 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:29.229283 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:14:29.230031 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:29.264225 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:29.275095 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:29.332223 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:29.366063 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:29.470928 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:29.536945 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:29.593458 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:29.645807 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:29.660966 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:29.727869 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:29.814018 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:29.836009 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:29.843689 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:29.873745 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:29.909041 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:14:29.962302 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:30.055307 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:30.076927 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.149461 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.232609 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.234241 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:30.395781 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.408831 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.412153 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:30.436547 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:30.452432 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:30.463778 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.585024 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:30.599480 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.709561 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:30.789525 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.806345 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:30.857860 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:30.924635 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:30.967216 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:30.969285 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:31.167234 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:31.333252 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:31.378225 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:31.379032 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:31.420902 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:31.530758 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:31.715840 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:31.721666 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:31.733574 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:31.760601 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:31.912197 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:31.966137 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:32.083993 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:32.115960 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:32.161823 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:32.208186 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:32.352104 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:32.353691 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:32.354162 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:32.357494 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:32.391026 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:32.411176 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:32.508198 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:32.521790 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:32.529464 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:14:32.565820 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:32.646987 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:32.768893 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:32.773808 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:32.806131 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:32.908201 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:32.958027 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:33.010264 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:33.010576 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:33.034500 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:33.084213 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:33.207501 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:33.208544 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:33.232620 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:33.292352 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:33.375766 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:33.422896 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:33.470286 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:33.478247 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:14:33.479084 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:33.503765 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:33.629531 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:33.636096 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:14:33.965087 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:34.002662 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:34.040611 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:34.077680 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:34.106245 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:34.242529 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:14:34.265447 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:34.377000 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:34.403752 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:34.416365 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:34.553095 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:34.573583 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:34.577381 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:34.579279 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:34.673348 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:34.786564 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:34.808704 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:34.819427 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:34.833245 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:34.942199 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:34.966076 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:35.009485 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:35.011062 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:14:35.029968 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.078593 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:35.128512 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.199961 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:35.226930 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:35.270190 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:35.336046 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.357570 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.447793 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.480360 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.501324 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:35.508647 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:35.574342 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.574926 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:35.673928 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.699767 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:35.753170 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.803067 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:35.828189 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:35.929477 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:35.952476 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:36.095181 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:36.151216 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:36.258009 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:36.320590 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(8f27eb42e75247c2aa142d3028e6ca2a): perf score=0.033394
W20240501 08:14:36.331741 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:36.374590 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(8f27eb42e75247c2aa142d3028e6ca2a) complete. Timing: real 0.053s	user 0.044s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":385,"dirs.queue_time_us":461,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":713,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":717,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":857,"threads_started":3,"wal-append.queue_time_us":378}
I20240501 08:14:36.376356 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(8f27eb42e75247c2aa142d3028e6ca2a): 4834 bytes on disk
I20240501 08:14:36.377257 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(8f27eb42e75247c2aa142d3028e6ca2a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 08:14:36.378082 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(bfbef7af4781448bb04b107b53d58ca0): perf score=0.033355
I20240501 08:14:36.445405 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(bfbef7af4781448bb04b107b53d58ca0) complete. Timing: real 0.067s	user 0.042s	sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1586,"drs_written":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":268,"threads_started":1,"wal-append.queue_time_us":354}
I20240501 08:14:36.446449 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0): 4921 bytes on disk
I20240501 08:14:36.446938 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240501 08:14:36.492758 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:36.526732 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:36.645666 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:36.699435 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(c665d7d20360440a9103e245f5f0900c): perf score=0.033379
I20240501 08:14:36.754186 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(c665d7d20360440a9103e245f5f0900c) complete. Timing: real 0.054s	user 0.040s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1805,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":946,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":227,"threads_started":1,"wal-append.queue_time_us":347}
I20240501 08:14:36.755750 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c): 4921 bytes on disk
I20240501 08:14:36.756994 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240501 08:14:36.772234 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:36.778939 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:36.835393 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(cf1f28d15c094fd98fac1b91d2785744): perf score=0.033347
I20240501 08:14:36.895927 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(cf1f28d15c094fd98fac1b91d2785744) complete. Timing: real 0.060s	user 0.038s	sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":391,"dirs.queue_time_us":2094,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":7027,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2473,"threads_started":3,"wal-append.queue_time_us":471}
I20240501 08:14:36.898061 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(cf1f28d15c094fd98fac1b91d2785744): 4834 bytes on disk
I20240501 08:14:36.898932 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(cf1f28d15c094fd98fac1b91d2785744) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240501 08:14:36.907759 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:36.913674 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:37.079759 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:14:37.151235 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(8f27eb42e75247c2aa142d3028e6ca2a): perf score=0.033386
W20240501 08:14:37.170997 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:37.195873 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:14:37.198073 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(8f27eb42e75247c2aa142d3028e6ca2a) complete. Timing: real 0.047s	user 0.040s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1448,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":695,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":607,"threads_started":1,"wal-append.queue_time_us":733}
I20240501 08:14:37.199450 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(8f27eb42e75247c2aa142d3028e6ca2a): 4834 bytes on disk
I20240501 08:14:37.200031 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(8f27eb42e75247c2aa142d3028e6ca2a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240501 08:14:37.228457 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:37.236177 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:37.380084 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:37.394537 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:37.397063 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:37.401820 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:37.524817 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:37.533552 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:37.539918 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:37.590881 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:37.597327 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:14:37.612421 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:14:37.702513 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(af24b8ab4d0c43ef986d89d4aefc263c): perf score=0.033335
I20240501 08:14:37.742750 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(af24b8ab4d0c43ef986d89d4aefc263c) complete. Timing: real 0.040s	user 0.026s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":372,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1308,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":474,"threads_started":2,"wal-append.queue_time_us":322}
W20240501 08:14:37.743534 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:37.744063 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(af24b8ab4d0c43ef986d89d4aefc263c): 4921 bytes on disk
I20240501 08:14:37.744635 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(af24b8ab4d0c43ef986d89d4aefc263c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 08:14:37.745541 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(d188d415f63f448195ae90e8e3d66e0e): perf score=0.033341
I20240501 08:14:37.761338 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(03d0f89e1d8f40fe84d275b117d8145f): perf score=0.033397
I20240501 08:14:37.792037 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(d188d415f63f448195ae90e8e3d66e0e) complete. Timing: real 0.046s	user 0.034s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":275,"threads_started":1,"wal-append.queue_time_us":389}
I20240501 08:14:37.793203 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(d188d415f63f448195ae90e8e3d66e0e): 4834 bytes on disk
I20240501 08:14:37.793828 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(d188d415f63f448195ae90e8e3d66e0e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 08:14:37.808965 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(03d0f89e1d8f40fe84d275b117d8145f) complete. Timing: real 0.047s	user 0.044s	sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1022,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1140,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1133,"threads_started":2,"wal-append.queue_time_us":310}
I20240501 08:14:37.810196 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(03d0f89e1d8f40fe84d275b117d8145f): 4921 bytes on disk
I20240501 08:14:37.810884 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(03d0f89e1d8f40fe84d275b117d8145f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
W20240501 08:14:37.875749 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:37.910593 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:37.934190 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:37.987828 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:37.999541 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:14:38.045637 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(92153666aebe47ddb080278cf24d1570): perf score=0.033370
I20240501 08:14:38.062763 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(ae946c165cd34853917e5c8a244b6408): perf score=0.033334
W20240501 08:14:38.075174 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:38.076983 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:14:38.084834 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(92153666aebe47ddb080278cf24d1570) complete. Timing: real 0.039s	user 0.021s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1171,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1277,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":730,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":232,"threads_started":1,"wal-append.queue_time_us":391}
I20240501 08:14:38.086138 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): perf score=0.033360
W20240501 08:14:38.093835 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:38.102830 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(ae946c165cd34853917e5c8a244b6408) complete. Timing: real 0.040s	user 0.029s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1182,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":231,"threads_started":1,"wal-append.queue_time_us":328}
I20240501 08:14:38.104113 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(ae946c165cd34853917e5c8a244b6408): 4582 bytes on disk
I20240501 08:14:38.104754 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(ae946c165cd34853917e5c8a244b6408) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:14:38.131440 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.044s	user 0.032s	sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":76,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1201,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":278,"threads_started":1,"wal-append.queue_time_us":437}
I20240501 08:14:38.132941 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(92153666aebe47ddb080278cf24d1570): 4921 bytes on disk
I20240501 08:14:38.133432 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(92153666aebe47ddb080278cf24d1570) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 08:14:38.134469 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): 4581 bytes on disk
I20240501 08:14:38.134949 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240501 08:14:38.218765 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:14:38.219907 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:38.273066 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:38.323809 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:38.364214 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:38.455574 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:38.464565 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:38.470094 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:38.512351 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:14:38.559891 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:38.563961 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:14:38.607442 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(850d4e1020e243f4b6fa113d2946846d): perf score=0.033354
I20240501 08:14:38.637509 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(ea502ae00eff4f72a74b8d25f5a11ede): perf score=0.033371
I20240501 08:14:38.648429 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(850d4e1020e243f4b6fa113d2946846d) complete. Timing: real 0.041s	user 0.037s	sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1371,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":629,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":540,"threads_started":2,"wal-append.queue_time_us":429}
I20240501 08:14:38.649667 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(850d4e1020e243f4b6fa113d2946846d): 4834 bytes on disk
I20240501 08:14:38.650135 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(850d4e1020e243f4b6fa113d2946846d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 08:14:38.670792 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(ea502ae00eff4f72a74b8d25f5a11ede) complete. Timing: real 0.033s	user 0.030s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":354,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":449,"threads_started":2,"wal-append.queue_time_us":296}
I20240501 08:14:38.671743 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(ea502ae00eff4f72a74b8d25f5a11ede): 4581 bytes on disk
I20240501 08:14:38.672207 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(ea502ae00eff4f72a74b8d25f5a11ede) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240501 08:14:38.687975 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:14:38.734261 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:38.837561 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:38.859444 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:38.928081 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:38.993301 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:39.083783 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:39.193545 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:39.200853 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:39.277619 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:39.356264 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:14:39.440667 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:39.511914 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:39.550240 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:39.601294 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:39.618914 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:39.625792 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:39.665812 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:39.769660 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:39.776183 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:39.812229 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:39.848192 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:39.866592 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:39.913297 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:14:39.926862 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(cd01dd912e1e4ca9a104270f3cd004b1): perf score=0.033335
I20240501 08:14:39.955919 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(cd01dd912e1e4ca9a104270f3cd004b1) complete. Timing: real 0.029s	user 0.023s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":443,"threads_started":2,"wal-append.queue_time_us":292}
I20240501 08:14:39.956964 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(cd01dd912e1e4ca9a104270f3cd004b1): 4579 bytes on disk
I20240501 08:14:39.957433 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(cd01dd912e1e4ca9a104270f3cd004b1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240501 08:14:40.005559 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:40.014601 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:40.055737 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:40.123385 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:40.259075 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:40.265621 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:14:40.278803 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:40.318706 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:40.357573 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:40.389262 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:40.478247 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:40.486389 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:40.503234 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:40.511984 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:40.525473 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:40.796727 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:40.812301 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:40.912297 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:40.950331 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:40.970108 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:41.043218 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:41.080060 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:41.123701 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:41.196619 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:41.206472 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:41.460834 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:41.504822 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:41.550828 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:41.577028 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:41.616703 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:41.620683 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:41.714677 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:42.037189 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:42.047482 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:42.074169 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:42.107106 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:42.256685 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:42.342576 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:42.351538 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:42.382736 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:42.405638 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:42.439298 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:14:42.466048 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(4b924c6344714a92ac37ae28961b02f5): perf score=0.033400
W20240501 08:14:42.480279 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:42.504609 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:42.506731 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:14:42.513310 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(4b924c6344714a92ac37ae28961b02f5) complete. Timing: real 0.047s	user 0.022s	sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":388,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":542,"threads_started":2,"wal-append.queue_time_us":364}
I20240501 08:14:42.514289 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(4b924c6344714a92ac37ae28961b02f5): 4581 bytes on disk
I20240501 08:14:42.514823 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(4b924c6344714a92ac37ae28961b02f5) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240501 08:14:42.565300 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:42.637071 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:42.655632 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:42.823235 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:42.849445 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:42.856567 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:14:42.915235 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(4799a93d96a64a75b1984bb85fcad7e8): perf score=0.033397
W20240501 08:14:42.934084 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:42.943354 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:42.956614 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(4799a93d96a64a75b1984bb85fcad7e8) complete. Timing: real 0.041s	user 0.037s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1084,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1556,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":994,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1231,"threads_started":2,"wal-append.queue_time_us":294}
I20240501 08:14:42.957911 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(4799a93d96a64a75b1984bb85fcad7e8): 4581 bytes on disk
I20240501 08:14:42.958509 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(4799a93d96a64a75b1984bb85fcad7e8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240501 08:14:42.958855 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:42.961324 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:43.005649 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:43.050832 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:14:43.084895 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:43.094676 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:43.248790 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:14:43.384927 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:43.437922 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:43.454675 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:14:43.461421 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(50b7150bbeb645d1b00d1a561c3276d1): perf score=0.033364
I20240501 08:14:43.521046 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(50b7150bbeb645d1b00d1a561c3276d1) complete. Timing: real 0.059s	user 0.050s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6284,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1353,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":730,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6620,"threads_started":2,"wal-append.queue_time_us":314}
I20240501 08:14:43.522469 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(cf1f28d15c094fd98fac1b91d2785744): perf score=0.033372
W20240501 08:14:43.523079 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:43.552557 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(cf1f28d15c094fd98fac1b91d2785744) complete. Timing: real 0.030s	user 0.022s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1369,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":222,"threads_started":1,"wal-append.queue_time_us":334}
I20240501 08:14:43.553581 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(50b7150bbeb645d1b00d1a561c3276d1): 4834 bytes on disk
I20240501 08:14:43.554087 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(50b7150bbeb645d1b00d1a561c3276d1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 08:14:43.555150 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(cf1f28d15c094fd98fac1b91d2785744): 4834 bytes on disk
I20240501 08:14:43.555648 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(cf1f28d15c094fd98fac1b91d2785744) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240501 08:14:43.557682 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:14:43.593578 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling FlushMRSOp(af24b8ab4d0c43ef986d89d4aefc263c): perf score=0.033341
W20240501 08:14:43.593816 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:14:43.624413 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: FlushMRSOp(af24b8ab4d0c43ef986d89d4aefc263c) complete. Timing: real 0.031s	user 0.022s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":311,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1231,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":392,"threads_started":2,"wal-append.queue_time_us":287}
I20240501 08:14:43.625861 10408 maintenance_manager.cc:419] P c3747ec2a93c4684962f215616e896ed: Scheduling UndoDeltaBlockGCOp(af24b8ab4d0c43ef986d89d4aefc263c): 4921 bytes on disk
I20240501 08:14:43.626471 10299 maintenance_manager.cc:643] P c3747ec2a93c4684962f215616e896ed: UndoDeltaBlockGCOp(af24b8ab4d0c43ef986d89d4aefc263c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240501 08:14:43.635479 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:43.653631 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:43.758270 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:14:43.769268 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(21065b7dcc18456eb1b4c2376d51eec4): perf score=0.033379
W20240501 08:14:43.775087 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:43.801334 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.032s	user 0.025s	sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1257,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":391,"threads_started":2,"wal-append.queue_time_us":259}
I20240501 08:14:43.802786 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4): 4921 bytes on disk
I20240501 08:14:43.803524 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 08:14:43.804845 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(f5a58b7166534844b0963f4fab46dfd9): perf score=0.033376
I20240501 08:14:43.846921 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(f5a58b7166534844b0963f4fab46dfd9) complete. Timing: real 0.042s	user 0.032s	sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1222,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":814,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":210,"threads_started":1,"wal-append.queue_time_us":307}
I20240501 08:14:43.847862 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(f5a58b7166534844b0963f4fab46dfd9): 4832 bytes on disk
I20240501 08:14:43.848423 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(f5a58b7166534844b0963f4fab46dfd9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240501 08:14:43.880580 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:14:43.893570 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:43.908210 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:44.051048 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:44.125612 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:44.137343 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:44.183457 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:44.243353 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:44.363046 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:44.390758 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:14:44.463826 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:44.608330 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:44.651011 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:44.686388 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:44.763227 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:44.813282 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:44.814447 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:44.834911 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:44.864173 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:44.881589 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:44.916230 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:44.943262 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:45.030840 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:45.033994 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:45.036324 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:45.055128 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:45.207211 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:45.208071 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:45.222692 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:45.231501 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:45.308781 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:45.318990 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:14:45.429752 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:45.482393 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:45.550129 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:45.594399 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:45.622603 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:45.695323 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:14:45.814554 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(bbef0147276d410da3d3830e682efe6d): perf score=0.033388
I20240501 08:14:45.845459 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(bbef0147276d410da3d3830e682efe6d) complete. Timing: real 0.031s	user 0.021s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":289,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1483,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":396,"threads_started":2,"wal-append.queue_time_us":308}
I20240501 08:14:45.846679 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(bbef0147276d410da3d3830e682efe6d): 4922 bytes on disk
I20240501 08:14:45.847249 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(bbef0147276d410da3d3830e682efe6d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 08:14:45.848593 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(d188d415f63f448195ae90e8e3d66e0e): perf score=0.033384
W20240501 08:14:45.850759 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:45.902781 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:14:45.918323 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(d188d415f63f448195ae90e8e3d66e0e) complete. Timing: real 0.070s	user 0.043s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1605,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":789,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10167,"threads_started":1,"wal-append.queue_time_us":10250}
I20240501 08:14:45.919723 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(6624441a1f1143648913595f905f8039): perf score=0.033393
I20240501 08:14:45.954188 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(6624441a1f1143648913595f905f8039) complete. Timing: real 0.034s	user 0.024s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1297,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":231,"threads_started":1,"wal-append.queue_time_us":341}
I20240501 08:14:45.955163 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(d188d415f63f448195ae90e8e3d66e0e): 4834 bytes on disk
I20240501 08:14:45.955621 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(d188d415f63f448195ae90e8e3d66e0e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 08:14:45.956429 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(6624441a1f1143648913595f905f8039): 4581 bytes on disk
I20240501 08:14:45.956882 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(6624441a1f1143648913595f905f8039) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 08:14:46.106750 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling FlushMRSOp(185a62881ea84131a0c14987e4296338): perf score=0.033399
W20240501 08:14:46.110939 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:46.133234 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:14:46.137621 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: FlushMRSOp(185a62881ea84131a0c14987e4296338) complete. Timing: real 0.031s	user 0.026s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1282,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":445,"threads_started":2,"wal-append.queue_time_us":332}
I20240501 08:14:46.138675 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338): 4919 bytes on disk
I20240501 08:14:46.139231 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240501 08:14:46.169826 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:46.193169 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:46.197929 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:46.273778 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:46.341878 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:46.389885 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:46.484081 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:46.489985 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:46.597654 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:46.667685 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:46.732676 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:46.806522 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:46.817282 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:47.057291 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:47.124606 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:47.135759 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:47.142163 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:47.260448 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:47.299437 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:47.380245 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:47.380658 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:47.426914 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:47.507164 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:47.510659 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:47.574927 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:47.633917 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:47.641428 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:47.642063 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:47.694097 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:47.711351 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:14:47.738943 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:47.838317 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:47.954003 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:47.992065 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:48.070122 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:48.103737 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:48.118041 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:48.157877 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:14:48.215404 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling FlushMRSOp(8a6ed4ea6d6941d2af0f81eaf744bcea): perf score=0.033383
W20240501 08:14:48.237097 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:14:48.256816 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: FlushMRSOp(8a6ed4ea6d6941d2af0f81eaf744bcea) complete. Timing: real 0.041s	user 0.022s	sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6466,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1299,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":793,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6832,"threads_started":2,"wal-append.queue_time_us":358}
I20240501 08:14:48.257828 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(8a6ed4ea6d6941d2af0f81eaf744bcea): 4921 bytes on disk
I20240501 08:14:48.258311 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(8a6ed4ea6d6941d2af0f81eaf744bcea) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 08:14:48.305842 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:48.341864 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:48.345134 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:48.581041 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:48.581734 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:48.584055 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:48.643311 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:14:48.702770 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:48.752070 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:48.818060 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:48.893718 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:48.894083 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:48.963877 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:14:48.964795 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:48.993530 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:48.998500 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:49.122401 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:49.135068 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:49.409845 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:14:49.410678 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:49.424351 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:49.446965 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:49.482316 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:49.519973 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:49.685652 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:49.727560 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:49.737053 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:49.772817 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:49.942806 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:49.951773 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:49.962947 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:49.998430 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.041074 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.077915 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:50.130654 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.139072 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:50.181622 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:50.227351 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:50.281352 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.289587 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:14:50.290524 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:50.410166 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:50.438280 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:50.493731 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:50.547065 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:14:50.569736 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.638386 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:50.730320 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.732369 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.747233 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:50.898806 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:50.948239 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:50.999526 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:51.018695 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:51.162463 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:51.175415 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:51.243507 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:51.402640 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:51.464399 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:51.492688 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:51.513952 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:51.557183 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:51.600826 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:51.814191 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:51.926826 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:51.980962 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:52.033483 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:52.094805 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:52.152034 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.155965 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.203043 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:52.205729 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:52.241190 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.249374 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.406975 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:52.429203 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:52.462806 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.539448 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:52.595775 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:52.667984 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:52.701639 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.752274 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.785090 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.796176 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:52.838404 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:52.851681 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:52.936435 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:53.052811 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:14:53.070338 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:53.150918 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:53.228605 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:53.247190 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:53.317791 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:53.337236 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:14:53.458778 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:53.501775 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:53.540206 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:53.587684 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:53.597690 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:53.688402 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:14:53.696089 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:53.728914 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:53.757437 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:53.849432 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:53.862257 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:53.863734 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:53.879703 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:54.022202 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:14:54.078086 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:54.098884 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:54.139801 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:54.255375 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:54.369385 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:54.416615 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:14:54.417443 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:54.558856 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:54.562230 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:54.574020 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:54.591818 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:54.592700 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:54.603667 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:54.650091 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:54.655776 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:54.799250 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:54.828462 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:54.983882 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.024052 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:55.086462 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:55.126235 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:55.143260 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.169202 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.190501 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:55.247745 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.254739 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:55.303438 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.442353 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:14:55.443475 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:55.568307 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:55.580807 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:55.623329 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:55.686297 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.687502 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:14:55.810032 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.860098 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:55.864830 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:55.969168 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:56.126042 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:56.164687 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:56.191941 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:56.268802 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:56.282836 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:56.342396 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:56.401429 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:56.440977 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:56.538890 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:56.545522 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:56.568148 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:56.600106 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:56.825471 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:56.919164 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:56.930953 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:56.991865 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:57.003526 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:57.047262 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:57.132594 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:57.169236 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:57.169984 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:57.240003 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:57.267709 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:57.397361 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:57.409471 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:57.452637 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:57.570466 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:57.670681 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:57.704217 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:57.828348 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:57.829996 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:57.899999 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:57.953022 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:58.008584 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:58.026878 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:58.036942 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:58.076970 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:58.156962 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:58.203301 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:58.263567 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:14:58.307550 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:58.378347 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:14:58.450984 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:58.481984 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:58.559568 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:14:58.640738 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:58.703083 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:14:58.715763 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:58.750655 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:58.752861 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:14:58.900758 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:58.992262 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:58.999228 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:59.006194 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:59.022349 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:14:59.023126 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:59.079735 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:59.183399 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:59.248692 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:59.318351 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:14:59.442152 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:59.474651 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:59.521550 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:59.550580 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:59.558552 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:14:59.559742 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:59.582574 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:14:59.692096 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:14:59.920318 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:14:59.923585 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:14:59.933104 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:14:59.942183 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:14:59.948668 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:14:59.985044 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:00.010578 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:00.185945 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:00.317929 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:00.328675 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:00.333405 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:00.369483 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:00.525434 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:00.526683 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:00.566663 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:00.584954 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:00.595939 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:00.735152 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:00.798827 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:00.800333 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:15:00.815297 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:00.823966 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:00.985560 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:01.016572 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:15:01.047784 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:01.104172 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:01.119843 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:01.261816 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:01.320706 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:01.378413 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:01.434548 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:01.492475 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:01.528002 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:01.556872 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:01.590003 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:01.621129 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:01.783210 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:01.832932 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:01.849444 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:01.939342 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:02.003672 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:02.086819 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:02.116593 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.120525 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.240064 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.309875 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.320953 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:02.432775 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:02.447263 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:02.541584 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:02.573161 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:02.629215 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.645213 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.645274 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.704098 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:02.825663 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:02.997558 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:03.018831 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:03.052402 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:03.129417 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:03.175282 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:03.220376 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:03.246989 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:03.282438 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:03.378355 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:15:03.399953 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:03.408185 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:03.510675 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:03.517738 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:03.547950 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:03.619746 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:03.655098 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:03.723891 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:04.003077 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:04.033775 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:04.054580 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:04.080437 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:04.086809 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:15:04.090080 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:04.139189 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:04.152547 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:04.157639 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:04.252177 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:04.462934 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:04.528062 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:04.532917 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:04.537822 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:04.583606 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:04.623836 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:04.625202 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:15:04.625979 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:04.630729 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:04.659339 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:04.838893 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:04.861447 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.013559 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:05.023525 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.036669 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:05.065330 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.085454 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:05.099996 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:05.113984 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:05.324407 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.334797 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.372711 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.442150 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:05.596529 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:05.597332 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:05.616861 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:05.671414 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:05.712584 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:05.739941 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.787408 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:05.831074 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:05.931375 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:15:06.015033 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:06.106365 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:06.178035 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:06.196782 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:06.224594 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:06.415582 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:06.427707 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:06.449872 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:06.479727 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:06.567704 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:06.625312 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:06.649178 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:06.653293 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:06.698120 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:06.813778 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:06.818024 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:06.969331 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.059794 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:07.073796 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:07.082391 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:07.105367 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:07.164403 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.175359 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.200822 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.230674 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:07.273756 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.603631 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:07.631567 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:07.698014 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:07.700496 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.751613 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.751667 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:07.766034 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:07.783942 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:07.892817 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:08.017408 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:08.177536 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:08.195712 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:08.209367 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:08.261682 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:08.267593 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:08.288692 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:08.324451 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:15:08.469341 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:08.556845 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:08.562512 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:08.625161 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:08.679216 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:08.788487 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:08.791635 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:08.836541 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:08.901266 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:08.948410 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:08.953032 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:08.957309 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:09.040879 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:09.052596 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:09.108671 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:15:09.109442 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:09.224752 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:09.309381 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:09.346755 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:09.422398 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:09.436687 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:09.634531 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:09.668623 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:09.692596 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:09.729871 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:15:09.730993 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:09.735183 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:09.845237 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:09.848444 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:09.848826 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:09.931911 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:09.965271 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:10.022663 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:10.073403 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:10.202237 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:10.346460 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:10.404580 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:10.405766 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:10.407711 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:10.431726 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:10.441347 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:10.465759 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:10.585758 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:10.635504 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:10.808964 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:10.917961 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:10.937570 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:10.938154 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:10.966885 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:15:11.019591 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:11.027832 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:11.222283 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:11.224350 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:11.235970 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:11.243657 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:11.300087 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:11.360612 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:11.443804 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:11.502588 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:11.509127 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:11.638531 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:11.684815 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:11.685124 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:11.757994 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:11.778224 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:11.916801 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:12.034602 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:12.057260 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:12.157143 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:12.189393 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:12.201184 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:12.209426 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:12.227900 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:12.251734 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:12.302454 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:12.386890 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:12.422497 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:12.516768 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:12.578536 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:12.781455 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:12.806355 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:12.911221 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:12.919204 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:12.920984 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:13.036684 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:13.037232 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:13.148128 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:13.245863 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:13.289252 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:13.339812 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:13.355772 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:15:13.424711 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:13.512260 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:13.560010 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:13.572166 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:15:13.663216 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:13.696264 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:13.801954 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:15:13.802783 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:13.833999 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:13.904578 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:13.974752 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:13.985477 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:13.993975 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:14.053879 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:14.130512 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:14.134533 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 08:15:14.142382 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:14.227589 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:14.289789 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:14.347898 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:14.483762 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:14.517663 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:14.628345 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:14.652730 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:14.655917 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:14.681712 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:14.722951 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:14.753598 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:14.841455 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:15:14.894094 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:14.909489 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:14.985152 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:15.002344 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:15.057736 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:15.098354 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:15.103538 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:15.107192 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:15.297104 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:15.456611 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:15.458873 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:15.508457 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:15.569633 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:15.650913 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:15.864966 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:15.876186 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:15.932654 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:15.949115 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:15.950851 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:15:15.975301 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:16.054347 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:16.090479 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:16.091630 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:16.400657 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:16.411164 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:16.434716 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:16.440508 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:16.467314 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:16.473218 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:16.504107 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:16.580961 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:16.653604 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:16.677953 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:16.689232 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:16.748484 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:16.814502 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:17.011080 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:17.041287 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:17.097883 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.130633 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:17.161387 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.231331 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.238415 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:17.273581 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:17.286078 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:17.287746 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:17.413548 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.426270 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.437885 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.576159 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:17.700261 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:17.821291 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:17.839634 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:17.884704 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.897162 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:17.897578 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:18.068475 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:18.163290 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:18.206161 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:18.390177 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:18.440649 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:18.486617 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:18.517900 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:15:18.522245 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:18.527046 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:18.600208 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:18.681097 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:15:18.771270 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:18.772228 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:18.829249 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:18.830046 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:18.847833 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:18.981787 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:18.995241 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:19.086498 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:19.100745 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:19.118927 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:19.220458 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:15:19.220777 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:19.263367 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:19.287860 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:19.370447 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:19.379669 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:19.517261 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:19.526018 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:19.602269 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:19.625016 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:19.739276 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:19.743324 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:19.818903 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:19.834673 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:19.847239 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 08:15:19.848208 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:19.894709 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:19.910064 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:20.016048 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:20.039399 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:20.121771 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:20.279865 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:20.296434 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:20.328948 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:20.342947 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:20.464830 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:20.510778 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:20.534588 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:20.758050 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:20.845351 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:20.863739 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:20.866029 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:20.866183 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:15:20.867643 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:20.936683 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:20.960464 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:21.163867 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:21.212213 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:21.285739 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:21.330333 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:21.377183 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:21.432262 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:21.438305 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:21.571451 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:21.589143 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:21.688846 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:21.691928 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:21.705929 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:21.793114 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:21.804344 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:21.830540 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:21.855947 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:21.965098 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:22.040391 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:22.063609 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.096890 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:22.144884 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.191932 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.204195 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.214672 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:22.215471 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:22.287245 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:22.430953 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:22.458062 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:22.481173 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.559484 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.610473 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:22.655251 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:22.714373 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.751673 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:22.863726 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:22.981062 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:22.988723 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:23.139335 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:23.153324 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:23.241909 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:23.297576 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:23.327088 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:23.331351 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:23.343812 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:15:23.429013 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:23.558902 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:23.659006 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:23.672031 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:23.690446 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:23.691344 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:23.792613 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:23.868374 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:15:23.870285 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:23.909855 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:24.080402 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:24.129947 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:24.190277 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:24.232980 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:24.253365 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:24.261126 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:24.276010 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:24.276892 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:24.330410 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:24.333720 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:24.374635 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:24.388191 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:24.408089 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:24.539291 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:24.585788 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:24.733915 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:24.753525 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:24.760471 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:24.806281 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:24.975389 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:15:24.976493 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:25.069851 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:25.138353 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:25.186012 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:25.268200 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:25.268981 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:25.271147 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:25.337172 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:25.396804 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:25.513509 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:25.550971 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:25.781630 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:25.805255 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:15:25.821055 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:25.852226 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:25.855649 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:25.860745 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:25.876392 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:25.947850 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:26.004130 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:26.264057 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:26.334650 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:26.445663 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:26.478842 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:26.517028 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:26.518679 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:26.577610 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:26.721014 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:26.731763 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:26.745566 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:26.835564 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:26.873859 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:26.875113 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:26.893544 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:26.926805 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:26.965273 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:26.984392 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:27.069006 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:27.130406 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:27.202363 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:27.219578 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:27.252192 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:27.373435 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:27.511454 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:27.514840 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:27.539012 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:27.564801 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:27.601128 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 08:15:27.601925 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:27.665781 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:27.682643 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:27.860786 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:27.920568 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:27.920861 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:28.034689 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:28.125731 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:28.241058 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:28.292654 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:15:28.303203 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:28.446497 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:28.464905 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:28.479940 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:28.536231 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:28.566162 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:28.636191 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:28.641554 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:28.766695 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:28.767627 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:28.795686 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:28.894395 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:28.918498 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:29.101197 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:29.139396 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:29.174811 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:29.256371 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:29.271368 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:29.302595 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:15:29.303529 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:29.333611 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:29.368242 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:29.378414 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:29.418730 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:29.466261 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:29.530819 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:29.550601 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:29.573117 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:29.712494 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:29.739362 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:29.787458 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:29.815521 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:29.979135 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:15:29.999703 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:30.031154 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:30.167260 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:30.185822 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:30.222575 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:30.239257 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:30.307888 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:30.450088 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:30.473500 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:30.477799 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:30.486517 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:30.719157 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:30.738165 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:30.794684 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:30.794787 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:15:31.054067 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:31.082679 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.085114 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.093963 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.246989 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:31.285573 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:31.321820 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.379725 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:31.398487 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:31.514185 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:31.606101 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.661238 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:31.686442 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.695161 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.729708 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:31.819223 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:31.828830 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:31.842116 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:31.893115 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.916301 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:31.985329 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:32.007972 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:32.008481 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:32.012344 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:32.040916 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:32.136375 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:32.173983 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:32.288386 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:32.315745 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:32.439360 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:32.474733 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:32.479269 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:32.480895 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:32.685133 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:32.717525 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:32.840592 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:32.910758 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:32.920979 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:32.965226 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:32.994326 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:32.995572 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:15:33.019330 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:33.056505 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:33.165557 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:33.327143 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:33.436489 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:33.501426 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:33.522544 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:33.526895 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:33.599638 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:33.624341 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:33.727035 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:33.796463 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:15:33.912384 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:33.917603 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:33.988569 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:15:33.998466 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:34.027199 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:34.067523 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:34.095377 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:34.098635 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:34.121855 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:34.169008 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:34.288564 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:34.299132 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:34.363276 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:15:34.471021 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:34.498950 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:34.548345 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:34.559005 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:34.600466 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:34.621054 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:34.632334 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:34.649704 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:34.663736 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:34.725878 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:34.880242 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:34.937422 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:34.953258 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:35.071398 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:35.178553 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:15:35.179628 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:35.220875 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:35.246096 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:35.311514 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:35.317521 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:35.404963 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:35.408068 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:15:35.423635 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:35.498080 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:35.575613 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:35.603754 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:35.844264 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:35.893734 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:35.903039 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:35.942536 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:35.997918 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:36.140437 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:36.211762 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:36.238373 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:36.382817 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:36.407850 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:36.411926 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:36.420678 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:36.437511 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:36.517746 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:36.532625 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:36.746130 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:36.821377 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:36.831264 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:36.833335 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:36.879611 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:36.975039 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:37.057607 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:37.071985 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:37.086562 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:37.090520 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:37.121634 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:37.211792 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:37.226347 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:37.262331 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:37.368042 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:37.386802 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:37.430941 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:37.579576 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:37.627790 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:37.640668 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:37.768168 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:37.786072 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:37.852344 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:37.985827 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:38.022131 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:38.031539 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:38.051614 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:38.052575 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:38.057215 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:38.079762 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:38.080641 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:15:38.339515 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:38.495909 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:38.496039 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:38.505178 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:38.539548 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:38.571501 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:38.740705 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:38.760066 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:38.762519 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:38.862998 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:38.907931 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:38.960592 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:38.972692 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:38.975054 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:39.048518 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:39.053598 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:39.173491 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:39.267035 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:39.325078 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:39.361292 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:39.376094 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:39.393894 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:39.476042 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:39.477172 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:39.494891 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:39.501972 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:39.550436 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:39.557725 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:39.641311 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:39.677253 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:39.783721 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:39.834178 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:39.840859 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:39.998353 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:40.018172 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:40.057749 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:40.209412 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:40.237571 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:40.335551 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:40.337950 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:15:40.521540 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:40.524927 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:40.535281 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:40.573601 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:40.629407 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:40.764808 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:40.771059 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:40.786780 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:40.893306 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:15:40.958714 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:41.049260 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:41.109090 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:41.111930 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:41.113596 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:41.143951 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:41.165822 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:41.188045 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:41.320052 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:41.356102 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:41.518307 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:41.546137 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:41.598538 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:41.666353 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:41.760828 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:41.776736 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:41.879520 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:41.981324 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:42.053944 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:42.078487 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:42.125408 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:42.132966 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:42.152493 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:42.164168 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:42.285336 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:42.309882 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:42.315659 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:42.330567 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:42.630061 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:42.687004 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:42.745537 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:42.764099 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:42.827003 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:42.855497 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:42.905334 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:42.907881 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:43.061941 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.171833 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:43.245512 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:43.267063 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.271734 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:43.291525 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.295608 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.404727 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:15:43.444913 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:43.446189 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:43.483429 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:43.637609 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:43.649498 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:43.755940 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.811242 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.819710 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:43.840391 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.931496 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:43.964910 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:43.997961 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:44.138175 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:44.148221 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:44.156929 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:44.232513 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:44.288664 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:44.316823 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:44.325630 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:44.422678 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:44.523285 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:44.534133 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:44.592399 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:44.619005 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:44.643246 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:44.673471 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:44.702860 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:44.731683 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:44.751179 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:44.763057 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:44.803361 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:44.934785 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:45.170291 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:45.217808 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:45.244868 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:45.296161 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:45.296245 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:45.335543 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:45.399767 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:45.485608 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:15:45.580662 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:45.603992 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:45.604146 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:45.685251 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:45.773366 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:45.833410 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:45.906843 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:45.917239 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:45.939814 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:15:46.037401 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:46.042196 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:46.063092 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:46.152524 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:46.209177 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:46.360661 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:46.430073 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:46.465461 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:46.468796 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:46.492357 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:46.586628 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:46.622289 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:46.654474 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:46.744680 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:46.792044 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:46.909171 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:46.921468 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:46.954908 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:47.015895 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:47.134860 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:47.139034 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:47.141500 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:47.165663 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:47.209223 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:47.312440 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:47.414496 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:47.449491 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:47.527493 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:47.543426 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:47.565261 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:47.576332 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:47.899453 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:47.944159 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:47.987766 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:47.990150 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:48.019251 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:48.036722 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:48.066907 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:48.081393 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:48.171650 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:48.246894 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:48.345748 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:48.392966 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:48.441903 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:48.481289 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:48.503234 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:48.503885 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:48.525646 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:48.550886 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:15:48.556537 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:48.817348 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:48.947139 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:48.948953 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:48.963491 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:48.964294 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:49.055399 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:49.090816 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:49.100781 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:49.150770 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:49.180785 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:49.293291 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:49.380779 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:49.406844 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:49.432528 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:49.497260 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:49.514814 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:49.523768 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:15:49.524554 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:49.628556 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:49.631410 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:49.713094 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:49.714488 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:49.883915 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:49.892416 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:49.899349 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:49.954246 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:50.033723 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:50.049105 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:50.083624 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:50.233351 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:50.287251 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:50.379035 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:50.489253 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:50.517078 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:50.548746 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:15:50.549571 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:50.635957 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:50.720527 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:50.820653 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:50.909353 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:50.913388 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:50.938477 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:50.976145 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:51.111172 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:51.118438 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:51.222429 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:51.280999 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:15:51.288853 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:51.324548 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:51.369792 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:51.441998 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:51.455253 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:51.522658 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:51.622191 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:51.643119 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:51.787794 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:51.825946 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:51.875476 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:51.912158 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:51.931663 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:51.960884 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:51.998044 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:52.083314 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:52.176810 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:52.253392 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:52.283442 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:52.344072 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:52.380964 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:52.383960 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:52.431557 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:52.485918 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:52.555723 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:52.639143 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:52.773712 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:52.791046 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:52.842653 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:52.921133 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:52.922789 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:53.037083 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:53.048272 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:53.225849 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:53.250336 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:53.338994 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:53.339921 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:15:53.350127 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:53.433372 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:53.467448 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:53.507673 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:53.589295 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:53.713253 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:53.750488 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:53.805446 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:15:53.819818 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:53.831609 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:53.857612 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:53.878623 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:53.879810 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:53.909085 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:54.011056 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:54.020507 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:54.025095 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:54.079085 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:54.193246 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:15:54.203156 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:54.263021 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:54.385948 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:54.445605 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:54.459674 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:54.470921 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:15:54.519853 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:54.520651 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:54.528033 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:54.658524 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:54.665912 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:54.669011 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:54.713338 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:54.722038 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:54.724571 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:54.764755 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:54.955130 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:55.021349 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:55.083968 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:55.215684 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:55.216400 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:55.265866 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:55.281561 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:55.306798 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:55.363298 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:55.520864 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:55.563499 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:15:55.598800 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:55.622992 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:55.784255 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:55.842658 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:55.864887 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:55.989871 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:56.054785 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:56.055393 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:56.160274 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:56.183517 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:56.240170 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:56.257658 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:56.259290 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:56.276439 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:15:56.277854 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:56.400354 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:56.430212 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:56.644348 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:56.666132 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:56.684409 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:56.705531 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:56.774398 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:56.777446 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:56.865825 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:56.985678 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:57.010820 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:57.081676 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:57.101140 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:57.155844 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:57.233150 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:57.243762 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:57.317546 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:57.442015 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:57.467046 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:57.467836 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:57.688530 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:57.717989 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:57.745842 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:57.769311 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:57.789583 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:57.809502 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:57.855114 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:57.929288 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:57.932384 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:58.032953 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:58.147042 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:58.367095 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:58.375609 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:58.424386 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:58.495949 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:58.511631 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:58.546638 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:58.759080 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:58.807564 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:58.830885 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:58.848191 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:58.854271 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:58.855266 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:15:58.901690 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:15:58.930984 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:58.983078 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:15:59.000305 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:59.150276 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:59.170311 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:59.226567 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:59.229035 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:59.229175 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:15:59.249774 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:59.313977 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:15:59.428486 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:59.460889 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:59.474712 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:59.573288 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:15:59.589601 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:15:59.608894 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:59.636291 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:15:59.640362 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:15:59.668995 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:59.821835 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:15:59.838487 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:15:59.871002 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:15:59.910152 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:15:59.991933 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:00.069833 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:00.197986 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:00.222580 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:00.237960 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:16:00.363960 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:00.396246 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:00.433550 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:00.478502 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:16:00.481374 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:00.491287 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:16:00.585575 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:00.662283 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:16:00.718322 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:00.759869 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:00.820297 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:00.833537 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:00.966410 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:01.104770 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:01.118508 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:01.133996 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:01.204484 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:01.267585 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:01.286378 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:01.364024 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:16:01.372932 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:01.443435 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:01.480600 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:01.639328 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:01.639596 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:01.682189 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:01.730293 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:01.887547 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:01.950134 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:02.049597 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:02.053220 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:02.093528 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:02.203641 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:02.210763 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:02.223292 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:02.250738 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:02.320408 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:02.382362 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:02.453970 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:02.533291 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:02.627132 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:02.645442 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:02.729909 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:02.758253 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:02.785182 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:02.882035 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:02.901584 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:03.067550 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:03.071836 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:03.128098 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:03.310979 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:03.311537 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:03.489048 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:03.520009 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:03.545935 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:03.551394 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:03.596437 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:03.622758 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:03.683409 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:03.744292 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:03.788239 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:16:03.884079 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:03.934365 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:04.020849 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:04.086585 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:04.140187 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:04.153781 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:04.154722 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:04.160542 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:04.170833 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:04.199452 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:04.253825 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:16:04.325979 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:04.419585 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:04.588694 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:04.599920 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:04.600741 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:04.612066 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:04.667045 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:04.687891 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:04.698433 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:04.701583 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:04.707453 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:04.807212 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:04.859882 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:04.869012 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:04.906272 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:16:04.990777 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:05.059836 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:05.152654 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:05.246681 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:05.283299 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:05.327770 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:05.365590 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:05.522311 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:05.633291 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:05.663726 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:16:05.664683 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:05.711891 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:05.727783 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:05.730621 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:05.768146 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:05.914901 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:05.930855 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:06.131925 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:06.143455 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:06.204140 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:06.247522 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:16:06.248739 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:06.273707 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:06.381053 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:06.423413 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:06.426995 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:06.476024 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:06.607553 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:06.686163 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:06.689827 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:06.751983 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:06.784706 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:06.796030 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:06.854084 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:06.931386 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:07.028754 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:07.066831 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:07.081828 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:07.216970 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:07.247491 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:07.262897 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:07.277709 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:07.349592 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:07.438918 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:07.502521 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:07.615310 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:07.627103 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:07.690498 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:07.773198 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:07.796146 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:07.836351 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:07.984210 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:08.082777 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:08.098095 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:08.116933 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:08.202376 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:08.245534 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:08.262652 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:08.289052 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:08.515997 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:08.611490 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:08.672412 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:08.673115 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:16:08.706046 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:08.721719 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:08.811666 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:08.844672 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:09.099434 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:09.111528 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:09.112828 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:09.126778 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:09.170454 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:09.194011 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:09.198611 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:09.314321 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:09.345332 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:16:09.346849 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:09.387430 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:09.392052 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:09.394893 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:09.546795 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:09.555545 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:09.600888 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:09.628624 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:09.662245 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:16:09.663480 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:09.684255 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:09.782446 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:09.828333 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:09.859263 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:09.879045 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:09.879897 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:10.009500 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:16:10.041561 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:10.132143 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:10.176965 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:10.253664 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:10.268728 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:10.372920 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:10.482998 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:10.649430 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:10.657562 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:10.735342 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:10.809753 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:10.872557 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:10.892711 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:16:10.921698 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:10.986970 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:11.026360 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:11.095324 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:11.135206 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:16:11.191242 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:11.217175 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:11.340458 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:11.389149 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:11.395067 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:11.487149 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:11.543180 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:11.609397 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:11.635213 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:11.701591 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:11.723066 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:11.736703 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:11.756186 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:11.803989 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:11.812007 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:11.839732 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:11.857699 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:11.876055 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:12.183178 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:12.202214 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:12.210363 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:12.299628 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:12.331045 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:12.367218 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:12.428854 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:12.434197 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:12.467113 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:12.607235 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:12.708015 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:12.822722 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:12.827678 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:12.979241 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:13.044780 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:13.046016 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:13.268762 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:13.326573 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:13.382881 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:13.425144 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:13.451056 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:13.495851 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:13.564950 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:13.588244 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:16:13.720717 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:13.851568 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:13.906508 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:13.918435 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:13.922505 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:13.945669 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:13.975261 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:13.986120 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:14.103003 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:14.111816 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:14.162753 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:14.184759 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:14.255450 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:14.275493 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:14.324328 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:14.363745 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:14.404213 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:14.466653 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:14.473666 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:14.524662 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:14.699031 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:16:14.699844 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:14.709714 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:14.733089 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:14.799294 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:14.868369 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:14.883690 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:15.029692 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:16:15.038153 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:15.067552 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:15.081344 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:15.105448 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:15.106721 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:15.118515 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:15.317102 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:15.317850 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:15.347712 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:15.577005 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:15.750072 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:15.798988 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:15.892530 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:15.905865 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:16:15.906814 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:15.935901 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:15.970983 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:16.055521 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:16.062593 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:16.083603 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:16:16.109841 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:16.242393 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:16.352311 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:16.374778 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:16.400307 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:16.430011 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:16.464622 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:16.512903 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:16.585527 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:16.622925 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:16.631572 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:16.648512 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:16.649397 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:16.731292 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:16.788106 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:16.819619 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:16.857924 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:16.861106 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:17.045815 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:17.082365 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:17.114151 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:17.199913 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:17.208068 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:17.387957 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:17.533147 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:17.565791 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:17.567270 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:17.599409 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:17.663022 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:17.664364 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:17.681181 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:17.935184 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:18.112565 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:18.222363 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:18.244422 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:18.262773 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:18.397951 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:18.409492 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:18.433943 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:18.526120 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:18.649919 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:18.660037 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:18.668756 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:18.715636 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:18.729817 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:18.805547 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:18.818404 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:18.833050 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:18.912681 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:18.914019 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:16:18.924400 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:18.962075 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:18.970070 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:19.049870 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:19.061561 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:19.153308 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:19.176201 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:19.231698 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:19.302516 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:19.405233 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:19.413188 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:19.544440 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:19.548002 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:19.591814 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:19.601091 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:19.649469 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:19.653637 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:19.738379 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:16:19.767989 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:19.891579 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:19.923022 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:19.966329 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:19.972445 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:20.064043 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:16:20.064968 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:20.124747 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:20.153331 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:20.159623 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:20.383953 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:20.443567 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:20.453761 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:20.604314 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:20.609784 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:20.672391 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:20.716297 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:20.836738 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:20.859908 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:20.903571 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:20.904424 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:20.930835 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:16:20.977072 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:20.994410 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:21.224758 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:21.233449 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:21.241539 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:21.269536 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:21.271287 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:21.317123 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:21.347205 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:21.385763 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:21.439754 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:21.473393 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:21.482440 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:21.614468 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:16:21.716953 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:21.758806 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:21.778329 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:21.786094 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:21.822638 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:21.894654 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:21.957535 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:21.988309 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:22.168172 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:22.173796 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:22.203331 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:22.211112 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:22.258932 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:22.326444 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:22.353144 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:22.392069 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:22.431608 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:22.642932 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:22.748854 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:22.789333 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:22.824010 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:22.902865 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:22.951768 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.181200 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.194056 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:23.222546 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:23.277256 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:23.402520 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:23.419103 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:23.426610 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.464164 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:23.499908 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.566218 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:23.587306 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:23.699137 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:23.776824 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.806366 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:23.820147 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.828368 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:23.835407 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.879240 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:23.882138 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:23.978401 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:24.013496 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:24.266355 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:24.353016 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:24.356685 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:24.388626 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:16:24.540930 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:24.541101 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:16:24.573987 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:24.626755 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:24.630195 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:24.645902 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:24.726718 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:24.735497 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:24.777818 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:24.816251 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:16:24.817011 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:24.897055 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:24.923085 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:25.038894 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:25.039480 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:25.102530 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:25.136235 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:25.148866 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:25.172555 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:25.333345 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:25.373418 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:25.520352 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:25.705713 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:25.714140 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:25.721673 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:25.766513 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:25.777773 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:25.835839 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:26.003247 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:26.009609 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:16:26.010529 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:26.074764 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:26.126044 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:26.151218 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:26.153614 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:26.202433 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:26.204623 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:26.208890 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:26.254990 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:26.257522 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:26.279697 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:26.378203 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:26.450029 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:26.493971 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:26.592940 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:26.625926 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:26.646757 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:27.007905 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:27.038950 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:27.069924 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:27.099383 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:27.132825 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:27.148567 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:27.243455 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:16:27.306470 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:27.308507 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:27.319944 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:27.332036 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:27.362707 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:27.472494 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:27.560106 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:27.583267 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:27.594772 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:27.628031 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:27.762653 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:27.794184 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:27.994669 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:28.052014 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:28.118469 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:28.148355 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:28.166476 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:28.187917 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:28.416368 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:28.417629 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:28.477567 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:28.515694 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:28.517410 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:28.596837 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:28.604424 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:28.643886 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:28.698407 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:28.917443 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:28.936378 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:28.946434 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:29.019075 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:29.032083 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:29.047714 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:29.056298 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:29.122704 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:29.133296 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:29.228382 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:29.422473 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:29.535759 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:29.538347 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:29.542141 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:16:29.606020 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:29.637709 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:29.686316 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:29.725100 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:29.744693 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:29.756196 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:16:29.816751 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:29.891630 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:16:30.066886 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:30.086436 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:30.127032 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:30.138960 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:30.188016 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:30.198343 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:16:30.199208 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:30.241839 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:30.388485 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:30.501752 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:30.567879 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:30.584285 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:30.667095 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:30.693833 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:30.797477 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:30.832106 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:30.885886 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:30.975195 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:30.985005 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:31.067520 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:31.106797 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:31.148736 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:16:31.177606 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:31.195423 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:31.197472 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:31.209445 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:31.238567 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:31.326912 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:31.374404 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:31.577836 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:31.591379 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:31.597672 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:31.667681 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:31.705857 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:31.742888 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:31.752280 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:31.755709 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:31.816856 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:32.128664 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:32.135319 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:32.152732 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:32.163822 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:32.246513 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:32.319054 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:16:32.328207 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:32.349471 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:32.459403 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:32.631906 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:32.666287 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:32.717849 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:32.731719 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:32.738031 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:32.780375 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:32.854686 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:32.857614 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:32.900319 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:33.018299 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:33.121690 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:33.125718 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:33.222307 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:33.234445 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:33.278331 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:33.539485 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:33.600978 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:33.662649 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:33.695842 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:33.701531 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:33.720360 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:33.725564 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:33.758473 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:33.801764 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:33.902268 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:33.931581 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:33.989893 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:34.060804 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:34.080395 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:34.148669 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:34.260097 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:34.271159 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:34.438380 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:34.443630 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:34.539566 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:34.550631 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 08:16:34.559746 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:34.637001 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:34.684224 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:34.704188 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:16:34.762373 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:34.836555 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:34.872602 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:34.898067 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:34.943814 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:34.996986 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:35.049762 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:35.106428 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:35.122983 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:35.148944 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:35.251250 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:35.304325 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:16:35.376024 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:35.427320 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:35.511754 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:35.520912 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:35.544658 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:35.626886 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:35.700464 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:35.726742 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:35.879542 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:35.933272 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:35.934374 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:35.965566 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:35.999465 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:36.184011 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:36.195979 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:36.229604 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:36.235193 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:36.247958 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:36.315879 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:36.353921 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:16:36.354842 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:36.357545 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:36.388520 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:36.433976 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:36.462491 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:36.527417 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:36.562834 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:36.687774 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:36.724314 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:36.857388 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:36.918335 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:36.976213 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:37.026573 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:37.106086 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:37.166301 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:37.197218 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:37.269869 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:16:37.343964 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:37.363282 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:37.384258 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:37.445379 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:37.491093 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:37.602655 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:37.682430 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:37.701496 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:37.740332 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:37.786629 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:37.852383 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:37.884680 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:37.928205 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:38.056095 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:38.065506 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:38.086356 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:38.254050 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:38.374912 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:38.400339 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:38.411269 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:38.428351 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:38.658282 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:38.681638 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:38.734649 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:38.776304 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:38.837045 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:38.863363 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:38.883664 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:38.985986 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:39.000558 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:39.031008 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:39.069852 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:39.240996 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:39.287743 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:39.310585 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:39.343713 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:39.414302 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:39.439227 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:39.455741 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:39.507266 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:39.508709 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:39.707407 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:16:39.789026 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:16:39.789809 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:39.795385 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:39.913628 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:39.920433 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:39.921564 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:39.974951 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:39.981034 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:16:40.075523 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:40.094422 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:40.193826 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:40.266721 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:40.335892 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:16:40.344620 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:40.367069 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:40.418599 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:40.425720 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:40.477257 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:40.494376 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:40.497419 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:40.759451 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:40.776240 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:40.793433 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:40.987473 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:41.019671 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:41.192814 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:41.202946 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:41.210165 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:41.245538 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:41.307835 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:41.314602 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:41.328742 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:41.334666 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:41.405107 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:41.525442 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:41.530195 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:41.549311 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:41.590888 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:41.648684 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:41.648754 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 08:16:41.741632 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:41.801283 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:41.856503 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:41.865429 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:41.929313 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:41.963390 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:41.970101 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:42.082475 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:42.162317 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:42.196394 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:42.261656 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:42.355518 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:42.498061 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:16:42.507897 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:42.641495 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:42.659763 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:42.670488 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:42.705232 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:42.722069 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:42.814827 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:42.832180 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:42.832366 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:42.839071 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:42.989842 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:43.185937 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:43.234138 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:43.278656 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:43.387329 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:43.447082 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:43.482607 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:43.508646 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:43.633550 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:43.717541 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:43.746115 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:43.788084 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:43.874176 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:43.881628 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:43.921895 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:43.941772 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:43.978993 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:44.060472 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:44.069802 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:44.092450 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:44.245988 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:44.253592 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:44.334012 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:44.379807 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:44.499035 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:44.503371 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:44.560488 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:44.608314 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:44.639688 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:44.796737 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:44.812695 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:16:44.813596 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:44.824312 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:16:44.842583 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:44.996503 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:16:45.095520 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:45.197086 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:45.198604 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:45.227464 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:45.228964 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:45.237375 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:45.251684 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:45.280247 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:45.301417 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:45.388926 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:45.566800 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:45.597949 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:45.622280 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:45.642786 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:45.724323 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:45.735473 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:45.939083 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:45.980917 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:46.050554 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:46.066249 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:46.125567 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:46.196722 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:46.218936 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:46.227538 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:46.254200 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:46.259624 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:46.266448 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:46.483971 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:46.503187 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:46.527165 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:46.533202 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:46.542171 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:46.608642 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:46.611959 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:46.659003 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:16:46.659706 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:46.773387 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:46.784411 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:46.890029 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:46.937830 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:46.957933 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:47.024150 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:47.073161 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:47.092721 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:47.213073 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:16:47.276500 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:47.322387 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:47.330080 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:47.452986 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:47.521998 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:47.615047 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:47.670842 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:47.675212 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:47.752820 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:47.818816 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:47.827886 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:47.844868 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:47.872635 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:48.101723 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:48.113667 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:48.163515 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:48.369897 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:48.404318 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:48.437722 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:48.438326 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:48.530791 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:48.574393 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:48.630455 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:48.640725 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:48.648321 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:48.755638 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:48.834852 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:48.836225 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:48.838322 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:48.872790 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:48.893570 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:48.907004 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:48.998327 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:49.080461 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:49.118429 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:49.336815 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:49.389524 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:49.444144 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:49.471179 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:49.509819 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:49.552170 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:49.584283 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:49.813215 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:16:49.814095 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:49.861729 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:49.874629 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:49.938097 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:16:50.016100 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:50.017009 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:50.031198 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:50.041025 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:16:50.041893 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:50.117075 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:50.357066 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:50.372537 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:50.397061 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:50.401788 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:50.411322 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:16:50.498970 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:50.510248 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:50.558845 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:50.711850 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:50.835140 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:50.843868 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:50.864076 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:50.919687 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:50.966044 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:51.051338 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:51.057825 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:51.152374 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:51.173302 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:51.257311 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:51.278725 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:51.291101 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:51.301512 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:51.356715 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:51.441819 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:51.464329 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:51.505690 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:51.509966 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:51.529243 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:51.586369 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:51.677242 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:16:51.745024 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:51.852528 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:51.866191 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:51.891336 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:51.979785 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:16:52.013963 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:52.064376 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:52.095865 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:52.308964 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:52.341080 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:52.396924 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:52.469705 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:52.481153 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:52.516932 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:52.522145 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:52.558835 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:16:52.765801 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:52.772444 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:52.839630 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:52.847501 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:52.865139 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:52.909675 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:52.921878 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:53.123903 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:53.148670 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:53.226369 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:53.256932 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:53.259604 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:53.391611 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:53.418839 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:53.434242 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:53.575860 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:53.609545 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:53.666510 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:53.711503 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:53.727556 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:53.734203 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:53.746923 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:53.808998 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:53.893105 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:53.907311 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:53.907655 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:53.908769 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6624441a1f1143648913595f905f8039: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:53.961410 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:54.022095 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:54.137185 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:54.170473 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:54.170917 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:54.288542 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:54.336089 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:54.444142 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:54.573550 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:54.618825 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:54.675915 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:54.705338 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:54.801731 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:54.833835 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:16:54.834592 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:54.928063 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:16:55.007404 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:55.015779 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:55.049036 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 08:16:55.049906 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:55.140846 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:55.174208 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:55.189767 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:55.280926 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:55.446394 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:16:55.447176 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:55.492412 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:55.522214 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:55.589653 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:55.590051 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:55.598759 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:55.687709 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:55.779985 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:55.796154 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:55.980806 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:55.986424 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:56.044060 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:56.119182 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:56.162106 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:56.183609 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:56.205307 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:56.242056 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:56.282466 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:56.285490 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:56.385881 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:56.408896 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:56.441443 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:56.506834 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:56.564209 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:56.704337 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:16:56.790757 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:56.809741 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:56.885787 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:56.936962 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:56.984454 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:56.995724 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:57.117511 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:57.200982 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:57.214102 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:57.283428 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:57.302985 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:57.535568 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:57.538764 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:16:57.544507 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:57.562625 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:57.564371 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:16:57.565681 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:16:57.603554 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:57.623371 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:57.714478 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:57.931691 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:57.939790 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:58.055088 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:58.116752 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:58.202895 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:58.258816 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:58.274633 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:58.297013 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:58.315933 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:16:58.334124 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:58.491937 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:58.501009 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:58.524533 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:58.570079 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:16:58.601538 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:58.617991 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:16:58.700661 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:58.710534 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:16:58.753219 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:58.911563 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:58.927520 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:58.972905 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:16:59.017043 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:59.081813 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:59.142355 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:59.282804 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:59.297526 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:59.341415 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:59.456468 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:16:59.504909 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:59.527134 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:59.655465 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:59.762869 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:16:59.793900 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:59.858321 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:16:59.899722 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:16:59.942451 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:16:59.996369 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:16:59.997085 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:00.097199 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 08:17:00.216356 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:00.230751 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:17:00.245332 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:17:00.258349 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:00.321331 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:17:00.337973 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:00.359000 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:00.459888 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:17:00.468861 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:00.729678 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:00.778949 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:00.797262 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:00.835829 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:00.838912 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:00.862287 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:00.895117 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:00.924629 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:01.078678 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:01.118722 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:01.152402 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:01.190977 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:01.267947 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:01.274713 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:01.296252 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:01.298209 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:01.298692 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:01.378541 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:01.400657 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:01.417066 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:01.628250 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:01.717214 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:01.721879 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:17:01.734931 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:17:01.896878 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:01.976253 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:01.990231 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:02.033972 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:17:02.034868 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:02.060899 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:02.122656 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:02.165352 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:17:02.288568 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:02.386229 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:02.386583 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:02.560956 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:02.652191 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:02.705662 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:02.757193 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:02.776993 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:17:02.817425 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:02.827617 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:02.938941 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:02.943764 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:03.022958 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:03.134833 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:03.173833 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.208073 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:03.389784 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.431303 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:03.479048 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:03.502089 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.558213 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:03.559034 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:03.566438 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:03.579773 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:03.702901 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.730926 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.759052 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.768855 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:03.816411 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:03.835189 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.844882 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:03.963742 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:03.989964 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:04.033936 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:04.094280 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:04.162287 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:04.204052 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:04.489794 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:04.509094 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:04.537446 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:04.552702 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:04.598970 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:04.644513 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:04.721216 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:17:04.747614 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:04.787358 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:05.062805 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:17:05.063591 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:05.096560 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:05.105747 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:05.115293 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:05.351390 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:05.382961 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:05.403259 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:05.508955 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:05.558863 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:17:05.613896 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:05.628068 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:05.638970 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:05.673130 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:05.796772 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:05.891683 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:05.951689 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:05.981988 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:05.984870 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:06.018023 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:06.089020 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:06.096573 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:06.142464 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:06.170886 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:06.212170 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:06.266324 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:06.372848 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:06.379410 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:06.389405 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:06.467847 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:06.479341 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:06.489593 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:06.647038 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:06.698581 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:06.704007 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:06.720214 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:06.870461 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:17:06.903637 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:06.907235 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:06.964582 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:07.091337 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:07.109699 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:17:07.209173 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:07.218040 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:07.252440 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:07.318475 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:17:07.419006 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:07.510950 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:07.518863 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:07.847808 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:07.849472 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:07.857769 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:07.956871 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:07.977532 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:07.987771 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:08.027989 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:08.248028 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:17:08.300630 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:08.301441 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:08.340005 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:08.376978 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:08.397583 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:08.404664 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:08.426651 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:08.444344 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:08.529824 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:08.580579 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:08.639693 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:08.674664 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:08.682932 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:08.729487 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:08.782833 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:08.793634 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:08.813174 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:08.931989 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:09.029320 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:09.057418 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:09.107622 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:09.168776 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:09.320353 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:09.357944 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:09.368248 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:09.383689 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:09.415990 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:09.572557 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:09.588464 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:09.714031 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:09.728088 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:09.752688 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:17:09.779165 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:09.885831 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:10.026074 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:10.077802 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:10.228163 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:17:10.229118 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:10.333446 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:10.338150 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:17:10.450588 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:10.541863 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:10.589500 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:10.590996 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:17:10.604990 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:10.622943 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:10.680887 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:10.740895 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:10.825448 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:10.867730 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:10.879216 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:10.971174 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:10.972095 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:10.980688 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:11.030390 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:11.047380 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:11.119061 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:11.128381 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:11.147701 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:11.147871 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:11.163106 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:11.194415 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:11.330967 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:11.344470 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:11.522189 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:11.566501 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:11.568706 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:11.643002 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:11.681623 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:11.804221 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:11.882756 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:11.914222 10298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:17:12.017946 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:12.101877 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:12.112179 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:12.169670 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:12.181227 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:12.243678 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:12.292018 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:12.363608 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:17:12.412340 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:12.434551 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:12.503423 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:12.749415 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:12.832664 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:12.841526 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:12.903429 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:12.986975 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:13.010977 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:13.093636 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:13.126224 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:13.147756 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:13.165452 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:13.364070 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:13.379863 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:13.393990 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:13.477046 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:13.480506 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:13.497622 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:13.531855 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:13.566105 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:13.580183 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:13.627769 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:13.653569 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:13.712944 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:13.729337 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:13.776855 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:13.881791 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:13.916940 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:13.954322 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:13.988408 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:14.003057 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:14.045500 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:14.303545 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:14.378728 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:14.388571 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:14.395242 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:14.434098 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:14.444990 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:14.538671 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:14.575071 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:14.717828 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:14.807643 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:14.974637 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:14.981091 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:15.023139 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:15.055824 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:17:15.059167 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:15.126163 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:15.374244 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:17:15.383101 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:15.398308 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:17:15.399577 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:15.475982 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:15.485311 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:15.490677 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:15.500340 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:15.561760 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:15.656734 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:15.726673 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:15.799625 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:17:15.842366 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:15.895865 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:15.913908 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:15.983422 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:15.990113 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:16.018023 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:16.021201 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:16.087273 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:16.096073 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:16.159792 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:16.201767 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:16.227727 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:16.366936 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:16.401779 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:16.448431 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:16.466183 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:16.512710 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:16.539887 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:16.753234 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:16.834439 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:16.883275 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:16.951602 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:17:16.952450 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:16.971264 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:16.999145 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:17.121316 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:17.268177 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:17.297802 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:17.387427 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:17.407011 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:17.426646 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:17.450405 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:17.516613 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:17.756316 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:17.782805 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:17.831912 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:17.858258 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:17.970844 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:17:18.005985 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:18.012845 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:18.021842 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:18.027125 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:18.079612 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:18.080789 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:18.115829 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:18.202906 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:18.425673 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:18.453490 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:18.495172 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:18.496490 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:18.575131 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:18.583617 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:18.611249 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:18.612095 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:18.657675 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:18.687453 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:18.712795 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:18.715744 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:18.788537 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:18.881690 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:18.933495 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:18.941383 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:19.107875 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:19.115100 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:19.190984 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:19.205060 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:19.404821 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:19.482821 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:19.516811 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:19.555665 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:19.584393 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:19.608783 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:19.647106 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:19.830541 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:19.851195 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:20.069547 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:20.111632 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:20.127736 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:20.169775 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:20.190732 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:20.346166 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:20.361348 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:17:20.397027 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:20.406222 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:17:20.406921 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:20.431324 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:17:20.541041 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:20.575215 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:20.605486 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:20.638840 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:20.668905 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:20.677412 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:20.833281 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:17:20.904201 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:20.935505 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:21.014261 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:21.082813 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:21.082813 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:21.108422 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:21.127182 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:21.154619 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:21.169651 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:21.242075 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:21.259795 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:21.340704 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:21.352954 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:21.440697 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:21.631155 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:21.668718 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:21.698626 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:21.725281 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:21.743397 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:21.996546 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:22.012424 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:17:22.013384 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:22.045133 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:22.133656 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:22.171937 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:22.176295 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:22.255710 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:22.404454 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:22.591645 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:22.617458 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:22.617565 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:22.624874 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:22.661136 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:22.682215 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:22.744930 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:22.762436 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:17:22.860522 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:22.887290 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:22.923995 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:22.984022 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:17:23.048463 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:23.102033 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:23.104416 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:23.202064 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:23.211772 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:23.272852 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:23.364362 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:23.580890 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:23.584367 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:23.600864 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:23.637030 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:23.646854 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:23.646849 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:23.654839 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:23.796623 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:23.832443 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:23.854712 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:23.966099 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:24.042816 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:24.046082 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:24.071216 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:24.123457 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:17:24.249058 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:24.347602 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:24.395038 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:24.438978 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:24.483814 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:24.574563 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:24.587237 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:24.755259 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:24.758558 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:24.791056 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:24.917825 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:24.965402 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:25.129889 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:25.138367 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:25.160789 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:17:25.197034 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:25.215451 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:25.306977 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:25.343220 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:25.385679 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:25.397266 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:25.403306 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:25.451929 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:17:25.466934 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:25.467617 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:25.581686 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:25.716538 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:25.755847 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:25.817528 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:25.911777 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:17:25.912691 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:25.963171 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:25.993228 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.091744 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.138321 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.206651 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:26.228549 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:26.238694 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:26.350805 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.396150 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:26.455106 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.488106 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:26.496052 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:26.632375 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.710819 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.713708 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:26.719722 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:26.784569 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:26.803274 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.941293 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:26.946795 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:26.987161 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:27.074928 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:17:27.075872 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:27.133087 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:27.231016 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:27.318408 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:27.376201 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:27.449443 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:27.611078 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:27.614931 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:27.674122 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:27.674235 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:27.686235 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:27.707906 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:27.761711 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:17:27.819761 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:27.850122 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:27.933871 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:28.019753 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:28.032864 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:28.125680 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:28.227309 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:28.239221 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:28.243542 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:28.282012 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:28.483479 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:28.540712 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:28.647868 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:28.662671 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:28.691310 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:28.736442 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:28.827877 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:28.905016 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:28.943732 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:29.001446 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:29.008203 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:29.033016 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:29.047287 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:29.060511 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:29.071012 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:29.179788 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:29.220965 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:29.389042 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:29.399428 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:29.503226 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:29.524921 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:29.554237 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:29.576516 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:29.656399 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:29.680385 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:29.683014 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:29.757272 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:29.777529 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:29.906137 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:29.952669 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:30.106328 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:30.147312 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:30.199921 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:30.242831 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:30.266047 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:30.284205 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:30.287022 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:30.358626 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:30.394554 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:17:30.488376 10688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:30.531806 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:17:30.592406 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:30.597425 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:30.678995 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:30.692364 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:30.719352 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:30.977104 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:17:30.977887 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:31.151723 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:31.151810 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:31.164638 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:31.182966 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:31.208370 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:31.231382 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:31.264142 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:31.371322 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:31.447237 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:31.602928 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:31.653203 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:31.655364 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:31.694618 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:31.751221 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:31.751369 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:31.765852 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:31.792915 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:31.801767 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:32.083655 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:32.097824 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:32.145037 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 08:17:32.146036 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:32.225951 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:32.302218 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:32.307940 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:32.322717 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:32.325387 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:32.359160 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:32.399283 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:32.517186 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:32.613682 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:32.647440 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:32.683666 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:32.832690 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:32.860204 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:32.886901 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:32.900298 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:33.034276 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:17:33.043514 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:33.049012 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:33.059077 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:33.106792 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:33.400171 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:33.424583 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:33.428407 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:33.484078 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:33.521631 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:33.582049 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:33.590036 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:33.612782 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:33.797039 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:33.804309 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:33.864377 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:33.873390 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:33.900563 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:34.192526 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:34.260298 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:34.274156 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:34.294174 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:34.334529 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:34.354806 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:34.365855 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:34.374161 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:34.402603 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:34.429451 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:34.482277 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:34.540349 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:34.590309 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:34.627169 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:34.712180 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:34.780215 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:34.804853 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:34.815649 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:34.834105 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:35.033504 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:35.082372 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:35.110731 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:35.161761 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:35.191699 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:35.212749 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:35.227890 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:35.320969 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:35.388809 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:35.530433 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:17:35.535640 10558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:17:35.575018 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:35.578332 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:35.609417 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:35.622712 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:35.684991 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:35.789355 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:35.871958 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:36.015866 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:17:36.034253 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:36.058893 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:36.141326 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:36.156687 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:36.215035 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:36.246831 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:36.261109 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:36.347060 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:36.412003 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:36.545522 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:36.582506 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:36.639473 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:36.649552 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:36.717445 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:36.797348 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:36.897779 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:36.946233 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:37.057977 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:37.059918 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:37.068379 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:37.070869 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:37.146270 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:17:37.147250 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:37.178774 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:37.232921 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:37.277354 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:37.279999 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:37.309024 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:37.386732 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:37.454921 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:37.466665 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:37.501693 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:37.609777 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:37.658252 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:37.690986 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:37.715431 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:37.745736 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:38.053140 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:38.058423 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:38.125905 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:38.204339 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:38.220161 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:38.263734 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:38.384196 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:17:38.508704 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:38.562801 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:38.599895 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:38.602927 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:38.624752 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:38.764928 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:38.775338 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:38.842895 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:38.857229 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:38.934274 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:17:38.955032 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:38.973237 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:39.112699 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:39.216614 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:39.265643 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:39.423861 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:39.499125 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:39.526643 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:39.529508 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:39.544234 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:39.566821 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:39.611526 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:39.615274 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:39.620474 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:39.767112 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:39.828631 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:39.831080 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:39.834451 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:39.851271 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:39.972148 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:39.976866 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:40.014657 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:40.051985 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:40.107793 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:40.118050 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:40.298935 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:40.371001 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:40.401989 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:40.452831 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:40.541795 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:40.591092 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:40.594230 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:40.712579 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:40.765359 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:40.769549 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:40.842466 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:17:40.854941 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:40.917148 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:41.086527 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:41.117733 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:41.118432 10428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:17:41.119151 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:41.121510 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:41.237993 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:41.303750 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:41.342945 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:41.361524 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:41.432413 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:41.537227 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:41.625886 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:41.747437 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:41.857076 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:41.915688 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:41.974773 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:42.001758 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:42.059064 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:42.061044 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:42.077430 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:42.151661 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:42.190467 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:42.209911 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:17:42.210988 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:42.217109 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:42.327787 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:42.361711 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:42.427150 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:42.528097 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:42.534057 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:42.570389 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:42.611646 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:42.613592 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:42.616936 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:42.656507 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:42.733048 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:42.750322 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:42.929054 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:43.026312 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:43.167721 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:43.206835 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:43.226449 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:43.477640 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:43.481590 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:17:43.497931 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:43.514052 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:43.521185 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:43.546911 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:43.605262 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:43.609046 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:43.725378 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:43.829151 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:43.831295 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:43.845276 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:44.078780 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:44.116963 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:44.198755 12603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:44.245832 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:44.429186 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:44.475802 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:44.553865 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:44.575870 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:44.589424 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:44.628515 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:44.629415 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:44.688740 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:44.755180 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:44.765628 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:44.862749 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:44.888938 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:17:44.960260 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:17:45.001953 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:45.024699 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:45.053278 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:45.094870 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:45.097414 12603 consensus_peers.cc:485] T bbef0147276d410da3d3830e682efe6d P c9f609481ded4d1aad9dd5f2341fd931 -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:45.149201 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:45.158048 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:45.233508 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:45.323321 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:17:45.379590 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:45.425762 24182 consensus_queue.cc:579] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.117s)
W20240501 08:17:45.428500 10558 consensus_peers.cc:462] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:17:45.437450 10621 consensus_queue.cc:237] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } }
I20240501 08:17:45.439715 10358 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:17:45.440320 24986 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:17:45.440446 10488 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:17:45.441025 24986 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:17:45.443164 24182 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } }
I20240501 08:17:45.443921 10358 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } }
I20240501 08:17:45.446178 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84789f1228f64be5b6976a9d52f64bd0 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:45.449059 10219 catalog_manager.cc:5484] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } }
W20240501 08:17:45.451792 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:17:45.460088 10492 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } }
W20240501 08:17:45.469949 10208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:17:45.470363 10621 consensus_queue.cc:237] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
W20240501 08:17:45.471065 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 84789f1228f64be5b6976a9d52f64bd0: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:17:45.472544 10492 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:17:45.473057 10358 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:17:45.472997 24986 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:17:45.473399 10557 consensus_peers.cc:485] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84789f1228f64be5b6976a9d52f64bd0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:45.473635 24986 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:17:45.475085 24985 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:45.475632 10492 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:45.476263 10358 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:45.480829 10219 catalog_manager.cc:5484] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 5 to 6, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:45.482417 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 84789f1228f64be5b6976a9d52f64bd0 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 08:17:45.584082 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:45.651444 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:45.670704 10298 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:17:45.688333 10687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:17:45.713644 10557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:17:45.714366 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:45.840162 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:45.856102 25044 ts_tablet_manager.cc:902] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:17:45.857290 25044 tablet_copy_client.cc:308] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:17:45.857936 10641 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 84789f1228f64be5b6976a9d52f64bd0 from peer c9f609481ded4d1aad9dd5f2341fd931 ({username='slave'} at 127.2.248.136:56369)
I20240501 08:17:45.858093 10641 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 84789f1228f64be5b6976a9d52f64bd0 from peer c9f609481ded4d1aad9dd5f2341fd931 at {username='slave'} at 127.2.248.136:56369: session id = c9f609481ded4d1aad9dd5f2341fd931-84789f1228f64be5b6976a9d52f64bd0
I20240501 08:17:45.860991 10641 tablet_copy_source_session.cc:205] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:45.862475 25044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84789f1228f64be5b6976a9d52f64bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:45.867240 25044 tablet_copy_client.cc:783] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:45.872386 25044 tablet_copy_client.cc:647] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:45.874729 25044 tablet_copy_client.cc:523] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:45.877442 25044 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap starting.
I20240501 08:17:45.890151 25044 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:45.890529 25044 tablet_bootstrap.cc:492] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: Bootstrap complete.
I20240501 08:17:45.891024 25044 ts_tablet_manager.cc:1367] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.003s
I20240501 08:17:45.891762 25044 raft_consensus.cc:348] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:17:45.891952 25044 raft_consensus.cc:729] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f609481ded4d1aad9dd5f2341fd931, State: Initialized, Role: LEARNER
I20240501 08:17:45.892150 25044 consensus_queue.cc:260] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:17:45.892616 25044 ts_tablet_manager.cc:1398] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240501 08:17:45.893294 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session c9f609481ded4d1aad9dd5f2341fd931-84789f1228f64be5b6976a9d52f64bd0 received from {username='slave'} at 127.2.248.136:56369
I20240501 08:17:45.893391 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session c9f609481ded4d1aad9dd5f2341fd931-84789f1228f64be5b6976a9d52f64bd0 on tablet 84789f1228f64be5b6976a9d52f64bd0 with peer c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:17:45.904055 12341 consensus_peers.cc:485] T cd01dd912e1e4ca9a104270f3cd004b1 P b6a964e54f3b44eda6d0666e0c1e4bea -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:17:45.966575 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:45.992385 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:17:46.018672 10428 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:46.099542 12713 maintenance_manager.cc:419] P c9f609481ded4d1aad9dd5f2341fd931: Scheduling UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0): 4581 bytes on disk
I20240501 08:17:46.100104 12604 maintenance_manager.cc:643] P c9f609481ded4d1aad9dd5f2341fd931: UndoDeltaBlockGCOp(84789f1228f64be5b6976a9d52f64bd0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240501 08:17:46.148813 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:46.159375 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:46.275993 24859 consensus_queue.cc:579] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.216s)
W20240501 08:17:46.277969 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:17:46.284570 10362 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:46.287187 12402 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 08:17:46.287508 10295 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:46.287787 25126 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:17:46.288153 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:46.290298 25124 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } } }
I20240501 08:17:46.290874 12402 raft_consensus.cc:2932] T 92153666aebe47ddb080278cf24d1570 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } } }
I20240501 08:17:46.293687 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 92153666aebe47ddb080278cf24d1570 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:46.294955 10221 catalog_manager.cc:5484] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 9 to 10, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:46.306143 24182 consensus_queue.cc:579] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.268s)
W20240501 08:17:46.308365 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:17:46.309933 10621 consensus_queue.cc:237] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:17:46.311729 10492 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:17:46.311774 12402 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:17:46.312249 25132 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:17:46.312516 25131 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:17:46.314322 10427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:17:46.315387 25132 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:46.316006 10492 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:46.317134 12402 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:46.320075 25131 consensus_queue.cc:579] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.197s)
I20240501 08:17:46.322710 10212 catalog_manager.cc:5484] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:17:46.325692 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea502ae00eff4f72a74b8d25f5a11ede with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:46.329517 10619 consensus_queue.cc:237] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
W20240501 08:17:46.332350 10557 consensus_peers.cc:485] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea502ae00eff4f72a74b8d25f5a11ede. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:46.334862 12402 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:46.335584 25137 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:46.336804 10488 raft_consensus.cc:1261] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:46.338639 10619 consensus_queue.cc:237] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:46.339732 25136 consensus_queue.cc:1035] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:46.341974 25137 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:46.343138 12402 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:17:46.346223 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:46.349136 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ea502ae00eff4f72a74b8d25f5a11ede with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:46.349431 10751 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:17:46.350135 25132 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:17:46.350636 10212 catalog_manager.cc:5484] T ea502ae00eff4f72a74b8d25f5a11ede P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:17:46.352499 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:46.354568 10555 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:46.354200 10488 raft_consensus.cc:2932] T ea502ae00eff4f72a74b8d25f5a11ede P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:46.365327 25132 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } } }
I20240501 08:17:46.365939 10751 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } } }
I20240501 08:17:46.368885 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aa6139020ac7458aa8a7ac08f63e6f26 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:46.370618 10221 catalog_manager.cc:5484] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:46.428560 12665 raft_consensus.cc:1203] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
I20240501 08:17:46.434652 25050 consensus_queue.cc:579] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.321s)
W20240501 08:17:46.436197 10428 consensus_peers.cc:462] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:17:46.443204 10488 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:17:46.445322 12665 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:17:46.445577 10751 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:17:46.445868 24218 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:17:46.446113 25141 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:17:46.448354 25050 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:46.448949 12665 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:46.451367 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c665d7d20360440a9103e245f5f0900c with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:46.453012 10221 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 11 to 12, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:17:46.451671 10751 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:46.458108 10488 consensus_queue.cc:237] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:17:46.460723 12667 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240501 08:17:46.461357 10425 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:46.461406 25141 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:17:46.463214 10749 raft_consensus.cc:1261] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:17:46.464730 25050 consensus_queue.cc:1035] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:17:46.465324 25141 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:46.466116 12667 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:46.469077 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c665d7d20360440a9103e245f5f0900c with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:46.468845 10749 raft_consensus.cc:2932] T c665d7d20360440a9103e245f5f0900c P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:46.470944 10221 catalog_manager.cc:5484] T c665d7d20360440a9103e245f5f0900c P c9f609481ded4d1aad9dd5f2341fd931 reported cstate change: config changed from index 12 to 13, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:17:46.488821 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c665d7d20360440a9103e245f5f0900c: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
W20240501 08:17:46.547778 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
W20240501 08:17:46.691037 10687 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:46.847962 25221 ts_tablet_manager.cc:902] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
I20240501 08:17:46.849583 25221 tablet_copy_client.cc:308] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:17:46.850333 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 92153666aebe47ddb080278cf24d1570 from peer 861f629e2d134ff08f25fa553dfb5bda ({username='slave'} at 127.2.248.130:44027)
I20240501 08:17:46.850455 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 92153666aebe47ddb080278cf24d1570 from peer 861f629e2d134ff08f25fa553dfb5bda at {username='slave'} at 127.2.248.130:44027: session id = 861f629e2d134ff08f25fa553dfb5bda-92153666aebe47ddb080278cf24d1570
I20240501 08:17:46.853435 10381 tablet_copy_source_session.cc:205] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:46.855062 25221 ts_tablet_manager.cc:1875] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:17:46.859695 25221 ts_tablet_manager.cc:1888] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 08:17:46.859905 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92153666aebe47ddb080278cf24d1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:46.862190 25221 tablet_copy_client.cc:783] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:46.862788 25226 ts_tablet_manager.cc:902] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:17:46.864101 25226 tablet_copy_client.cc:308] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:17:46.864773 10641 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet aa6139020ac7458aa8a7ac08f63e6f26 from peer 861f629e2d134ff08f25fa553dfb5bda ({username='slave'} at 127.2.248.130:36253)
I20240501 08:17:46.864939 10641 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet aa6139020ac7458aa8a7ac08f63e6f26 from peer 861f629e2d134ff08f25fa553dfb5bda at {username='slave'} at 127.2.248.130:36253: session id = 861f629e2d134ff08f25fa553dfb5bda-aa6139020ac7458aa8a7ac08f63e6f26
I20240501 08:17:46.865182 25088 consensus_queue.cc:579] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.511s)
I20240501 08:17:46.867974 10641 tablet_copy_source_session.cc:205] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:46.869773 25221 tablet_copy_client.cc:647] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:46.870432 25226 ts_tablet_manager.cc:1875] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:17:46.871654 25221 tablet_copy_client.cc:523] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:46.875952 25221 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:17:46.875980 10749 consensus_queue.cc:237] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:17:46.878832 10362 raft_consensus.cc:1261] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:17:46.879097 10687 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b72323b8b99f4fd1aedcce69089f6204. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:46.879376 25088 consensus_queue.cc:1035] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:17:46.880076 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:46.883708 25226 ts_tablet_manager.cc:1888] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:17:46.884021 25226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6139020ac7458aa8a7ac08f63e6f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:46.892903 10362 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:46.896708 25231 raft_consensus.cc:2932] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:46.897905 25226 tablet_copy_client.cc:783] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:46.899278 10212 catalog_manager.cc:5484] T b72323b8b99f4fd1aedcce69089f6204 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:46.904525 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b72323b8b99f4fd1aedcce69089f6204 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:46.911209 25236 ts_tablet_manager.cc:902] T ea502ae00eff4f72a74b8d25f5a11ede P c9f609481ded4d1aad9dd5f2341fd931: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
F20240501 08:17:46.911651 25236 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.queue_time_us, tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.queue_time_us) (tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_cpu_time_us, tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_cpu_time_us) (tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_wall_time_us, tablet-download-pool-03d0f89e1d8f40fe84d275b117d8145f.run_wall_time_us) (tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.queue_time_us, tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.queue_time_us) (tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_cpu_time_us, tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_cpu_time_us) (tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_wall_time_us, tablet-download-pool-4799a93d96a64a75b1984bb85fcad7e8.run_wall_time_us) (tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.queue_time_us, tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.queue_time_us) (tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_cpu_time_us, tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_cpu_time_us) (tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_wall_time_us, tablet-download-pool-50b7150bbeb645d1b00d1a561c3276d1.run_wall_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.queue_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.queue_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.run_cpu_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.run_cpu_time_us) (tablet-download-pool-6624441a1f1143648913595f905f8039.run_wall_time_us, tablet-download-pool-6624441a1f1143648913595f905f8039.run_wall_time_us) (tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.queue_time_us, tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.queue_time_us) (tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.run_cpu_time_us, tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.run_cpu_time_us) (tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.run_wall_time_us, tablet-download-pool-84789f1228f64be5b6976a9d52f64bd0.run_wall_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_cpu_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_cpu_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_wall_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.run_wall_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.queue_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.queue_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_cpu_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_cpu_time_us) (tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_wall_time_us, tablet-download-pool-8a6ed4ea6d6941d2af0f81eaf744bcea.run_wall_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.queue_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.queue_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_cpu_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_cpu_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_wall_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_wall_time_us) (tablet-download-pool-ae946c165cd34853917e5c8a244b6408.queue_time_us, tablet-download-pool-ae946c165cd34853917e5c8a244b6408.queue_time_us) (tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_cpu_time_us, tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_cpu_time_us) (tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_wall_time_us, tablet-download-pool-ae946c165cd34853917e5c8a244b6408.run_wall_time_us) (tablet-download-pool-bbef0147276d410da3d3830e682efe6d.queue_time_us, tablet-download-pool-bbef0147276d410da3d3830e682efe6d.queue_time_us) (tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_cpu_time_us, tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_cpu_time_us) (tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_wall_time_us, tablet-download-pool-bbef0147276d410da3d3830e682efe6d.run_wall_time_us) (tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.queue_time_us, tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.queue_time_us) (tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_cpu_time_us, tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_cpu_time_us) (tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_wall_time_us, tablet-download-pool-bfbef7af4781448bb04b107b53d58ca0.run_wall_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.queue_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.queue_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_cpu_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_cpu_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_wall_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_wall_time_us) (tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.queue_time_us, tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.queue_time_us) (tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_cpu_time_us, tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_cpu_time_us) (tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_wall_time_us, tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_wall_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.queue_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.queue_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_cpu_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_cpu_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_wall_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_wall_time_us) (tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.queue_time_us, tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 08:17:46.913852 25226 tablet_copy_client.cc:647] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:46.915652 25226 tablet_copy_client.cc:523] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7f96a8ee7dcd  google::LogMessage::Fail() at ??:0
I20240501 08:17:46.920058 25226 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
    @     0x7f96a8eebb93  google::LogMessage::SendToLog() at ??:0
    @     0x7f96a8ee77cc  google::LogMessage::Flush() at ??:0
    @     0x7f96a8ee8f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:17:46.927414 25221 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:46.927919 25221 tablet_bootstrap.cc:492] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap complete.
I20240501 08:17:46.928289 25221 ts_tablet_manager.cc:1367] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.052s	user 0.027s	sys 0.009s
I20240501 08:17:46.929155 25221 raft_consensus.cc:348] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:46.929473 25221 raft_consensus.cc:729] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: LEARNER
    @     0x7f96abc28742  kudu::TraceMetrics::InternName() at ??:0
I20240501 08:17:46.929703 25221 consensus_queue.cc:260] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:46.930583 25221 ts_tablet_manager.cc:1398] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:17:46.931490 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session 861f629e2d134ff08f25fa553dfb5bda-92153666aebe47ddb080278cf24d1570 received from {username='slave'} at 127.2.248.130:44027
I20240501 08:17:46.931620 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session 861f629e2d134ff08f25fa553dfb5bda-92153666aebe47ddb080278cf24d1570 on tablet 92153666aebe47ddb080278cf24d1570 with peer 861f629e2d134ff08f25fa553dfb5bda
    @     0x7f96abbf4f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f96abbeecd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f96bb0418de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f96bb059daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f96bb184480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f96bb1bca8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:17:46.958380 25137 raft_consensus.cc:1050] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 to VOTER
    @     0x7f96bb19f981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 08:17:46.959372 25137 consensus_queue.cc:237] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:17:46.963290 25226 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:46.963560 25226 tablet_bootstrap.cc:492] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap complete.
I20240501 08:17:46.963846 25226 ts_tablet_manager.cc:1367] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.044s	user 0.033s	sys 0.007s
I20240501 08:17:46.964447 25226 raft_consensus.cc:348] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:46.964661 25226 raft_consensus.cc:729] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: LEARNER
I20240501 08:17:46.964824 25226 consensus_queue.cc:260] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:46.965910 25226 ts_tablet_manager.cc:1398] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
    @     0x7f96bb19f70e  std::_Function_handler<>::_M_invoke() at ??:0
I20240501 08:17:46.966516 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session 861f629e2d134ff08f25fa553dfb5bda-aa6139020ac7458aa8a7ac08f63e6f26 received from {username='slave'} at 127.2.248.130:36253
I20240501 08:17:46.966637 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session 861f629e2d134ff08f25fa553dfb5bda-aa6139020ac7458aa8a7ac08f63e6f26 on tablet aa6139020ac7458aa8a7ac08f63e6f26 with peer 861f629e2d134ff08f25fa553dfb5bda
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 08:17:46.973402 10362 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:17:46.973428 12667 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P c9f609481ded4d1aad9dd5f2341fd931 [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
    @     0x7f96abbf8b75  kudu::ThreadPool::DispatchThread() at ??:0
I20240501 08:17:46.974002 25139 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:17:46.974323 25243 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
    @     0x7f96abbfc19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 08:17:46.981691 10488 raft_consensus.cc:1261] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
    @     0x7f96abbfc019  std::_Function_handler<>::_M_invoke() at ??:0
I20240501 08:17:46.982726 25139 consensus_queue.cc:1035] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f96abbd701e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f96aef5c6db  start_thread at ??:0
    @     0x7f96a7d9a71f  clone at ??:0
I20240501 08:17:46.998106 25139 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:46.998587 10362 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:47.003767 10488 raft_consensus.cc:2932] T 84789f1228f64be5b6976a9d52f64bd0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:47.004781 10219 catalog_manager.cc:5484] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 6 to 7, c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:47.029598 25139 consensus_queue.cc:579] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.133s)
W20240501 08:17:47.035888 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:17:47.036780 10619 consensus_queue.cc:237] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:47.040633 10555 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b6f09b0b5e84329944360d511578ed0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.041769 10488 raft_consensus.cc:1261] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:47.042420 25139 consensus_queue.cc:1035] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:17:47.047641 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.049870 25265 ts_tablet_manager.cc:902] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:17:47.050716 25139 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:47.055099 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b6f09b0b5e84329944360d511578ed0 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:47.055963 10492 raft_consensus.cc:2932] T 9b6f09b0b5e84329944360d511578ed0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:17:47.060663 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:17:47.065589 10212 catalog_manager.cc:5484] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 8 to 9, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:47.066407 25220 consensus_queue.cc:579] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.184s)
W20240501 08:17:47.071781 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:47.072786 10362 consensus_queue.cc:237] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:47.091369 10295 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 850d4e1020e243f4b6fa113d2946846d. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.091730 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.093909 25265 tablet_copy_client.cc:308] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:17:47.100728 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet c665d7d20360440a9103e245f5f0900c from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:47839)
I20240501 08:17:47.100889 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet c665d7d20360440a9103e245f5f0900c from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:47839: session id = b6a964e54f3b44eda6d0666e0c1e4bea-c665d7d20360440a9103e245f5f0900c
I20240501 08:17:47.104056 10512 tablet_copy_source_session.cc:205] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:47.105641 25265 ts_tablet_manager.cc:1875] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:17:47.114041 25265 ts_tablet_manager.cc:1888] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 08:17:47.114307 25265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c665d7d20360440a9103e245f5f0900c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:47.120931 25265 tablet_copy_client.cc:783] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:47.124743 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(92153666aebe47ddb080278cf24d1570): 4921 bytes on disk
I20240501 08:17:47.125370 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(92153666aebe47ddb080278cf24d1570) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 08:17:47.126794 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(aa6139020ac7458aa8a7ac08f63e6f26): 4834 bytes on disk
I20240501 08:17:47.127264 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(aa6139020ac7458aa8a7ac08f63e6f26) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 08:17:47.127453 25265 tablet_copy_client.cc:647] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:47.129102 25265 tablet_copy_client.cc:523] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:47.134079 25265 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap starting.
I20240501 08:17:47.138145 25232 consensus_queue.cc:579] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.060s)
I20240501 08:17:47.142796 10749 consensus_queue.cc:237] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:17:47.145866 10488 raft_consensus.cc:1261] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:17:47.146450 25231 consensus_queue.cc:1035] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:17:47.149740 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.162678 25265 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:47.162442 25231 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:47.163064 25265 tablet_bootstrap.cc:492] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Bootstrap complete.
I20240501 08:17:47.163327 25265 ts_tablet_manager.cc:1367] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent bootstrapping tablet: real 0.029s	user 0.021s	sys 0.001s
I20240501 08:17:47.164007 25265 raft_consensus.cc:348] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:17:47.164283 25265 raft_consensus.cc:729] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6a964e54f3b44eda6d0666e0c1e4bea, State: Initialized, Role: LEARNER
I20240501 08:17:47.164471 25265 consensus_queue.cc:260] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:17:47.165143 25265 ts_tablet_manager.cc:1398] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240501 08:17:47.165930 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-c665d7d20360440a9103e245f5f0900c received from {username='slave'} at 127.2.248.134:47839
I20240501 08:17:47.166054 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session b6a964e54f3b44eda6d0666e0c1e4bea-c665d7d20360440a9103e245f5f0900c on tablet c665d7d20360440a9103e245f5f0900c with peer b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:17:47.167706 10492 raft_consensus.cc:2932] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:47.172366 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a63e16f11894c2886091dcd3d204106 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:47.172799 10221 catalog_manager.cc:5484] T 2a63e16f11894c2886091dcd3d204106 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 6 to 7, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
W20240501 08:17:47.179154 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:47.194211 25231 consensus_queue.cc:579] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.246s)
I20240501 08:17:47.198091 10749 consensus_queue.cc:237] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:47.202364 10685 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50b7150bbeb645d1b00d1a561c3276d1. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.203028 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.220801 12451 maintenance_manager.cc:419] P b6a964e54f3b44eda6d0666e0c1e4bea: Scheduling UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c): 4921 bytes on disk
I20240501 08:17:47.221400 12342 maintenance_manager.cc:643] P b6a964e54f3b44eda6d0666e0c1e4bea: UndoDeltaBlockGCOp(c665d7d20360440a9103e245f5f0900c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 08:17:47.273211 10488 raft_consensus.cc:1203] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20240501 08:17:47.273783 10297 connection.cc:535] client connection to 127.2.248.136:42153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:47.273917 10297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 52 similar messages]
W20240501 08:17:47.274948 10297 consensus_peers.cc:485] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.275110 10687 connection.cc:535] client connection to 127.2.248.136:42153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:47.275215 10557 connection.cc:535] client connection to 127.2.248.136:42153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:47.276144 10427 connection.cc:535] client connection to 127.2.248.136:42153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:47.277348 10427 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.277467 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.278005 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:47.278110 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.278203 10557 consensus_peers.cc:485] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.278299 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.278409 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.281088 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.283970 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:17:47.286553 10427 consensus_peers.cc:485] T 4799a93d96a64a75b1984bb85fcad7e8 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.302361 24859 consensus_queue.cc:579] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.118s)
I20240501 08:17:47.306340 10362 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:17:47.308619 10751 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:17:47.309481 24859 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:17:47.309859 10427 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.310776 10297 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.311992 25220 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:47.312587 10751 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:47.314918 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae946c165cd34853917e5c8a244b6408 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:47.316550 10221 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:47.321506 10362 consensus_queue.cc:237] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:47.324234 10295 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae946c165cd34853917e5c8a244b6408. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.324587 10751 raft_consensus.cc:1261] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 08:17:47.324986 10297 consensus_peers.cc:485] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.325100 10297 consensus_peers.cc:485] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.325592 25220 consensus_queue.cc:1035] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:17:47.327728 24859 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:47.331530 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae946c165cd34853917e5c8a244b6408 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:47.333317 10488 raft_consensus.cc:1203] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240501 08:17:47.333401 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:47.334026 10751 raft_consensus.cc:2932] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:47.341591 10212 catalog_manager.cc:5484] T ae946c165cd34853917e5c8a244b6408 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 11 to 12, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:17:47.358991 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:47.361267 25137 consensus_queue.cc:579] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.107s)
W20240501 08:17:47.363055 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:17:47.365579 10619 consensus_queue.cc:237] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } }
W20240501 08:17:47.376524 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): Couldn't send request to peer 06c1439c9ab8415485eff20566811ede. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03d0f89e1d8f40fe84d275b117d8145f. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.378468 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.378625 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.403206 25050 consensus_queue.cc:579] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.550s)
I20240501 08:17:47.404268 25232 consensus_queue.cc:579] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.354s)
I20240501 08:17:47.409003 10751 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } }
I20240501 08:17:47.410531 10488 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:17:47.414858 10619 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:17:47.415478 25050 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:17:47.418329 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.419986 25050 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:47.424363 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bfbef7af4781448bb04b107b53d58ca0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:47.427037 10619 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:47.428063 10217 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:47.430240 12403 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 08:17:47.431015 25234 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:17:47.431661 10362 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 08:17:47.432291 25231 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:17:47.436043 10488 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } }
W20240501 08:17:47.437328 10427 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:47.448808 10621 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:47.450810 25350 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:47.453780 25050 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } } }
I20240501 08:17:47.457984 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bfbef7af4781448bb04b107b53d58ca0 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:47.460920 10619 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } } }
W20240501 08:17:47.463297 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:17:47.463965 10217 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 8 to 9, NON_VOTER 06c1439c9ab8415485eff20566811ede (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:47.481632 25231 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
W20240501 08:17:47.485275 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.133:43981, user_credentials={real_user=slave}} blocked reactor thread for 47970.5us
I20240501 08:17:47.486855 10362 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
I20240501 08:17:47.487461 12403 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } }
I20240501 08:17:47.490185 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af24b8ab4d0c43ef986d89d4aefc263c with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:47.491644 10217 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 14 to 15, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:17:47.493402 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093): Couldn't send request to peer 06c1439c9ab8415485eff20566811ede. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfbef7af4781448bb04b107b53d58ca0. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:47.494207 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.499629 10751 consensus_queue.cc:237] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
I20240501 08:17:47.502050 10362 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20240501 08:17:47.504051 10208 catalog_manager.cc:4573] TS 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae946c165cd34853917e5c8a244b6408: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:17:47.505051 25231 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 08:17:47.505632 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.505775 12403 raft_consensus.cc:1261] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 08:17:47.506323 25231 consensus_queue.cc:1035] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 08:17:47.507644 25234 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:47.508145 10358 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:47.510578 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af24b8ab4d0c43ef986d89d4aefc263c with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:47.510854 10732 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:17:47.514175 10217 catalog_manager.cc:5484] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 15 to 16, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:47.517225 12403 raft_consensus.cc:2932] T af24b8ab4d0c43ef986d89d4aefc263c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:47.518573 10600 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:47.549003 10472 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:47.552016 25265 ts_tablet_manager.cc:902] T 9b6f09b0b5e84329944360d511578ed0 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:17:47.565773 12385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:47.578783 10558 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:17:47.579505 12402 raft_consensus.cc:1203] T c665d7d20360440a9103e245f5f0900c P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
W20240501 08:17:47.583324 10557 consensus_peers.cc:485] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.586241 25360 ts_tablet_manager.cc:902] T 850d4e1020e243f4b6fa113d2946846d P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
F20240501 08:17:47.586683 25360 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-185a62881ea84131a0c14987e4296338.queue_time_us, tablet-download-pool-185a62881ea84131a0c14987e4296338.queue_time_us) (tablet-download-pool-185a62881ea84131a0c14987e4296338.run_cpu_time_us, tablet-download-pool-185a62881ea84131a0c14987e4296338.run_cpu_time_us) (tablet-download-pool-185a62881ea84131a0c14987e4296338.run_wall_time_us, tablet-download-pool-185a62881ea84131a0c14987e4296338.run_wall_time_us) (tablet-download-pool-21065b7dcc18456eb1b4c2376d51eec4.queue_time_us, tablet-download-pool-21065b7dcc18456eb1b4c2376d51eec4.queue_time_us) (tablet-download-pool-21065b7dcc18456eb1b4c2376d51eec4.run_cpu_time_us, tablet-download-pool-21065b7dcc18456eb1b4c2376d51eec4.run_cpu_time_us) (tablet-download-pool-21065b7dcc18456eb1b4c2376d51eec4.run_wall_time_us, tablet-download-pool-21065b7dcc18456eb1b4c2376d51eec4.run_wall_time_us) (tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.queue_time_us, tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.queue_time_us) (tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_cpu_time_us, tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_cpu_time_us) (tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_wall_time_us, tablet-download-pool-4b924c6344714a92ac37ae28961b02f5.run_wall_time_us) (tablet-download-pool-6e56f5f3cd9f4a56a3d43543ac0e5d14.queue_time_us, tablet-download-pool-6e56f5f3cd9f4a56a3d43543ac0e5d14.queue_time_us) (tablet-download-pool-6e56f5f3cd9f4a56a3d43543ac0e5d14.run_cpu_time_us, tablet-download-pool-6e56f5f3cd9f4a56a3d43543ac0e5d14.run_cpu_time_us) (tablet-download-pool-6e56f5f3cd9f4a56a3d43543ac0e5d14.run_wall_time_us, tablet-download-pool-6e56f5f3cd9f4a56a3d43543ac0e5d14.run_wall_time_us) (tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us, tablet-download-pool-850d4e1020e243f4b6fa113d2946846d.queue_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.queue_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.queue_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_cpu_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_cpu_time_us) (tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_wall_time_us, tablet-download-pool-8f27eb42e75247c2aa142d3028e6ca2a.run_wall_time_us) (tablet-download-pool-92153666aebe47ddb080278cf24d1570.queue_time_us, tablet-download-pool-92153666aebe47ddb080278cf24d1570.queue_time_us) (tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_cpu_time_us, tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_cpu_time_us) (tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_wall_time_us, tablet-download-pool-92153666aebe47ddb080278cf24d1570.run_wall_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.queue_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.queue_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_cpu_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_cpu_time_us) (tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_wall_time_us, tablet-download-pool-9b6f09b0b5e84329944360d511578ed0.run_wall_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.queue_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.queue_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_cpu_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_cpu_time_us) (tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_wall_time_us, tablet-download-pool-af24b8ab4d0c43ef986d89d4aefc263c.run_wall_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.queue_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.queue_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_cpu_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_cpu_time_us) (tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_wall_time_us, tablet-download-pool-c665d7d20360440a9103e245f5f0900c.run_wall_time_us) (tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.queue_time_us, tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.queue_time_us) (tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_cpu_time_us, tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_cpu_time_us) (tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_wall_time_us, tablet-download-pool-cd01dd912e1e4ca9a104270f3cd004b1.run_wall_time_us) (tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.queue_time_us, tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.queue_time_us) (tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_cpu_time_us, tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_cpu_time_us) (tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_wall_time_us, tablet-download-pool-cf1f28d15c094fd98fac1b91d2785744.run_wall_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.queue_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.queue_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_cpu_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_cpu_time_us) (tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_wall_time_us, tablet-download-pool-d188d415f63f448195ae90e8e3d66e0e.run_wall_time_us) (tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.queue_time_us, tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.queue_time_us) (tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.run_cpu_time_us, tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.run_cpu_time_us) (tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.run_wall_time_us, tablet-download-pool-ea502ae00eff4f72a74b8d25f5a11ede.run_wall_time_us) (tablet-download-pool-f5a58b7166534844b0963f4fab46dfd9.queue_time_us, tablet-download-pool-f5a58b7166534844b0963f4fab46dfd9.queue_time_us) (tablet-download-pool-f5a58b7166534844b0963f4fab46dfd9.run_cpu_time_us, tablet-download-pool-f5a58b7166534844b0963f4fab46dfd9.run_cpu_time_us) (tablet-download-pool-f5a58b7166534844b0963f4fab46dfd9.run_wall_time_us, tablet-download-pool-f5a58b7166534844b0963f4fab46dfd9.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 08:17:47.594533 24218 consensus_queue.cc:579] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.407s)
    @     0x7ff798a89dcd  google::LogMessage::Fail() at ??:0
I20240501 08:17:47.597393 10341 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:47.600569 10488 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
    @     0x7ff798a8db93  google::LogMessage::SendToLog() at ??:0
I20240501 08:17:47.612820 10751 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
    @     0x7ff798a897cc  google::LogMessage::Flush() at ??:0
I20240501 08:17:47.613476 25350 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:47.615782 12402 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:47.616372 25350 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
    @     0x7ff798a8af59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:17:47.622537 25350 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:47.625473 12403 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
    @     0x7ff79b7ca742  kudu::TraceMetrics::InternName() at ??:0
I20240501 08:17:47.630964 10217 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea reported cstate change: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:47.634423 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 185a62881ea84131a0c14987e4296338 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:47.638156 10488 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
W20240501 08:17:47.642727 10425 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): Couldn't send request to peer 8d5f781e3ba94cbfbbea9b5782ef8eaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 185a62881ea84131a0c14987e4296338. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.643718 12403 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:47.644237 24218 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:17:47.646639 24218 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } } }
I20240501 08:17:47.650242 12405 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P b6a964e54f3b44eda6d0666e0c1e4bea [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } } }
I20240501 08:17:47.652318 25369 ts_tablet_manager.cc:902] T 50b7150bbeb645d1b00d1a561c3276d1 P b6a964e54f3b44eda6d0666e0c1e4bea: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:17:47.653497 25322 consensus_queue.cc:579] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.423s)
    @     0x7ff79b796f8b  kudu::ThreadPool::ThreadPool() at ??:0
W20240501 08:17:47.655191 10297 consensus_peers.cc:462] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
    @     0x7ff79b790cd6  kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 08:17:47.658629 10358 consensus_queue.cc:237] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } }
I20240501 08:17:47.660413 25265 tablet_copy_client.cc:308] T 9b6f09b0b5e84329944360d511578ed0 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
    @     0x7ff7aabe38de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7ff7aabfbdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 08:17:47.668901 10641 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 9b6f09b0b5e84329944360d511578ed0 from peer b6a964e54f3b44eda6d0666e0c1e4bea ({username='slave'} at 127.2.248.134:41459)
I20240501 08:17:47.669092 10641 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 9b6f09b0b5e84329944360d511578ed0 from peer b6a964e54f3b44eda6d0666e0c1e4bea at {username='slave'} at 127.2.248.134:41459: session id = b6a964e54f3b44eda6d0666e0c1e4bea-9b6f09b0b5e84329944360d511578ed0
    @     0x7ff7aad26480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:17:47.675451 10641 tablet_copy_source_session.cc:205] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:47.676957 25265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6f09b0b5e84329944360d511578ed0. 1 dirs total, 0 dirs full, 0 dirs failed
    @     0x7ff7aad5ea8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:17:47.686501 10751 raft_consensus.cc:1261] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:17:47.686568 10619 raft_consensus.cc:1261] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:17:47.687163 25220 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:17:47.687463 10751 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:47.687528 25271 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:17:47.688280 25050 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
    @     0x7ff7aad41981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 08:17:47.689373 10749 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:47.693395 25265 tablet_copy_client.cc:783] T 9b6f09b0b5e84329944360d511578ed0 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:47.693828 10749 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } } }
    @     0x7ff7aad4170e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7ff79b79ab75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7ff79b79e19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7ff79b79e019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240501 08:17:47.713286 25265 tablet_copy_client.cc:647] T 9b6f09b0b5e84329944360d511578ed0 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Starting download of 1 WAL segments...
    @     0x7ff79b77901e  kudu::Thread::SuperviseThread() at ??:0
I20240501 08:17:47.718618 25265 tablet_copy_client.cc:523] T 9b6f09b0b5e84329944360d511578ed0 P b6a964e54f3b44eda6d0666e0c1e4bea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7ff79eafe6db  start_thread at ??:0
    @     0x7ff79793c71f  clone at ??:0
I20240501 08:17:47.724689 25220 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } }
I20240501 08:17:47.726817 10619 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } }
I20240501 08:17:47.731681 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e99538651cf64510a811ba01c6c8fb40 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:47.732697 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 185a62881ea84131a0c14987e4296338 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:47.732445 10217 catalog_manager.cc:5484] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240501 08:17:47.734966 10751 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } }
I20240501 08:17:47.736768 10218 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 9 to 10, NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } } }
I20240501 08:17:47.741742 10358 consensus_queue.cc:237] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:17:47.746712 10619 raft_consensus.cc:1261] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:17:47.748696 25271 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:17:47.751049 25322 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:47.752043 10751 raft_consensus.cc:1261] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:17:47.754688 24859 consensus_queue.cc:1035] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:17:47.755224 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e99538651cf64510a811ba01c6c8fb40 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:47.756014 10619 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:47.756477 10751 raft_consensus.cc:2932] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:47.761889 10218 catalog_manager.cc:5484] T e99538651cf64510a811ba01c6c8fb40 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 5 to 6, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
W20240501 08:17:47.817800 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:47.819799 25137 consensus_queue.cc:579] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.003s)
W20240501 08:17:47.822168 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:17:47.830600 25137 raft_consensus.cc:1050] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: attempting to promote NON_VOTER 861f629e2d134ff08f25fa553dfb5bda to VOTER
I20240501 08:17:47.831306 25137 consensus_queue.cc:237] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } }
I20240501 08:17:47.833734 10751 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:17:47.834174 10488 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:17:47.834311 25139 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:17:47.834784 25139 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:17:47.836025 10558 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.841264 25137 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } } }
I20240501 08:17:47.841804 10488 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } } }
I20240501 08:17:47.846060 10751 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } } }
I20240501 08:17:47.846455 10218 catalog_manager.cc:5484] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 7 to 8, 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:17:47.855259 10619 consensus_queue.cc:237] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } }
I20240501 08:17:47.857254 10751 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:47.857913 10488 raft_consensus.cc:1261] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:47.857901 25137 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:47.858497 25137 consensus_queue.cc:1035] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:47.859791 25243 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } } }
I20240501 08:17:47.860337 10751 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } } }
I20240501 08:17:47.863165 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa6139020ac7458aa8a7ac08f63e6f26 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:47.864969 10221 catalog_manager.cc:5484] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:17:47.865226 10488 raft_consensus.cc:2932] T aa6139020ac7458aa8a7ac08f63e6f26 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } } }
I20240501 08:17:47.874233 25322 raft_consensus.cc:1050] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed: attempting to promote NON_VOTER 861f629e2d134ff08f25fa553dfb5bda to VOTER
I20240501 08:17:47.874951 25322 consensus_queue.cc:237] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false } }
I20240501 08:17:47.876762 10488 raft_consensus.cc:1261] T 92153666aebe47ddb080278cf24d1570 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 08:17:47.878110 10298 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:47.880491 25220 consensus_queue.cc:1035] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:17:47.947333 25400 ts_tablet_manager.cc:902] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: Initiating tablet copy from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319)
I20240501 08:17:47.948776 25400 tablet_copy_client.cc:308] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34319
I20240501 08:17:47.949729 10641 tablet_copy_service.cc:140] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Received BeginTabletCopySession request for tablet 03d0f89e1d8f40fe84d275b117d8145f from peer 06c1439c9ab8415485eff20566811ede ({username='slave'} at 127.2.248.132:56417)
I20240501 08:17:47.949898 10641 tablet_copy_service.cc:161] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Beginning new tablet copy session on tablet 03d0f89e1d8f40fe84d275b117d8145f from peer 06c1439c9ab8415485eff20566811ede at {username='slave'} at 127.2.248.132:56417: session id = 06c1439c9ab8415485eff20566811ede-03d0f89e1d8f40fe84d275b117d8145f
I20240501 08:17:47.953557 10641 tablet_copy_source_session.cc:205] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:47.955310 25400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d0f89e1d8f40fe84d275b117d8145f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:47.960551 25405 ts_tablet_manager.cc:902] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:17:47.964183 25400 tablet_copy_client.cc:783] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:47.969863 25400 tablet_copy_client.cc:647] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:47.971539 25400 tablet_copy_client.cc:523] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:47.974881 25405 tablet_copy_client.cc:308] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:17:47.975169 25400 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
I20240501 08:17:47.975817 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet bfbef7af4781448bb04b107b53d58ca0 from peer 06c1439c9ab8415485eff20566811ede ({username='slave'} at 127.2.248.132:41085)
I20240501 08:17:47.975994 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet bfbef7af4781448bb04b107b53d58ca0 from peer 06c1439c9ab8415485eff20566811ede at {username='slave'} at 127.2.248.132:41085: session id = 06c1439c9ab8415485eff20566811ede-bfbef7af4781448bb04b107b53d58ca0
W20240501 08:17:47.977207 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:47.978631 10512 tablet_copy_source_session.cc:205] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:47.980545 25405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfbef7af4781448bb04b107b53d58ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:47.980888 25137 consensus_queue.cc:579] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.171s)
I20240501 08:17:47.987332 25405 tablet_copy_client.cc:783] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:47.990377 10619 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:17:47.996538 10751 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:17:47.997131 25139 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:17:47.998064 25405 tablet_copy_client.cc:647] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: tablet copy: Starting download of 1 WAL segments...
W20240501 08:17:48.000389 10555 connection.cc:535] client connection to 127.2.248.134:44821 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:48.001111 10685 connection.cc:535] client connection to 127.2.248.134:44821 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:17:48.002573 25405 tablet_copy_client.cc:523] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:17:48.002739 10295 connection.cc:535] client connection to 127.2.248.134:44821 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:48.004959 25410 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:17:48.001356 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.134:44821 (local address 127.2.248.131:54297)
0501 08:17:48.001623 (+   267us) negotiation.cc:107] Waiting for socket to connect
0501 08:17:48.001641 (+    18us) client_negotiation.cc:174] Beginning negotiation
0501 08:17:48.001769 (+   128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:17:48.004385 (+  2616us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":170}
I20240501 08:17:48.007820 25405 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: Bootstrap starting.
W20240501 08:17:48.008396 10555 consensus_peers.cc:485] T cf1f28d15c094fd98fac1b91d2785744 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.008507 10555 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.008584 10555 consensus_peers.cc:485] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.014493 25139 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:48.018342 10749 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:48.019461 25400 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:17:48.019932 25400 tablet_bootstrap.cc:492] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: Bootstrap complete.
I20240501 08:17:48.020208 25400 ts_tablet_manager.cc:1367] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.045s	user 0.022s	sys 0.009s
I20240501 08:17:48.020867 25400 raft_consensus.cc:348] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } }
I20240501 08:17:48.021263 25400 raft_consensus.cc:729] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: LEARNER
I20240501 08:17:48.021467 25400 consensus_queue.cc:260] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "bb2f0099fb794400b5c24927e213e5d8" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 43981 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } }
I20240501 08:17:48.022905 25400 ts_tablet_manager.cc:1398] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:17:48.024238 10641 tablet_copy_service.cc:342] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Request end of tablet copy session 06c1439c9ab8415485eff20566811ede-03d0f89e1d8f40fe84d275b117d8145f received from {username='slave'} at 127.2.248.132:56417
I20240501 08:17:48.024323 10641 tablet_copy_service.cc:433] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: ending tablet copy session 06c1439c9ab8415485eff20566811ede-03d0f89e1d8f40fe84d275b117d8145f on tablet 03d0f89e1d8f40fe84d275b117d8145f with peer 06c1439c9ab8415485eff20566811ede
I20240501 08:17:48.025492 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b924c6344714a92ac37ae28961b02f5 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 08:17:48.026892 21600 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:17:48.001901 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.134:44821 (local address 127.2.248.132:56635)
0501 08:17:48.003618 (+  1717us) negotiation.cc:107] Waiting for socket to connect
0501 08:17:48.003637 (+    19us) client_negotiation.cc:174] Beginning negotiation
0501 08:17:48.003765 (+   128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:17:48.026181 (+ 22416us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1592}
W20240501 08:17:48.027158 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:48.027263 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.028563 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:17:48.002157 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.134:44821 (local address 127.2.248.130:49393)
0501 08:17:48.002385 (+   228us) negotiation.cc:107] Waiting for socket to connect
0501 08:17:48.002402 (+    17us) client_negotiation.cc:174] Beginning negotiation
0501 08:17:48.002511 (+   109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:17:48.027863 (+ 25352us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132}
W20240501 08:17:48.029122 10425 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.029001 10221 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 10 to 11, NON_VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 08:17:48.030227 10425 consensus_peers.cc:485] T c665d7d20360440a9103e245f5f0900c P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.031291 14507 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:17:48.003535 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.134:44821 (local address 127.2.248.129:45491)
0501 08:17:48.003684 (+   149us) negotiation.cc:107] Waiting for socket to connect
0501 08:17:48.003712 (+    28us) client_negotiation.cc:174] Beginning negotiation
0501 08:17:48.003823 (+   111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:17:48.030751 (+ 26928us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
W20240501 08:17:48.031767 10295 consensus_peers.cc:462] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:48.031885 10295 consensus_peers.cc:462] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:48.031996 10295 consensus_peers.cc:485] T e99538651cf64510a811ba01c6c8fb40 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.032101 10295 consensus_peers.cc:485] T 92153666aebe47ddb080278cf24d1570 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.032227 10295 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.034297 10619 consensus_queue.cc:237] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } }
W20240501 08:17:48.038031 10555 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.038161 10557 consensus_peers.cc:485] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.043260 10297 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:17:48.044488 10750 raft_consensus.cc:1261] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 8d5f781e3ba94cbfbbea9b5782ef8eaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:17:48.045327 25139 consensus_queue.cc:1035] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:17:48.047763 25137 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:48.048899 10749 raft_consensus.cc:2932] T 4b924c6344714a92ac37ae28961b02f5 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } } }
I20240501 08:17:48.051887 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4b924c6344714a92ac37ae28961b02f5 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:48.054394 10221 catalog_manager.cc:5484] T 4b924c6344714a92ac37ae28961b02f5 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 11 to 12, NON_VOTER c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136) added. New cstate: current_term: 1 leader_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: NON_VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.065601 25405 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:48.065939 25405 tablet_bootstrap.cc:492] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: Bootstrap complete.
I20240501 08:17:48.066159 25405 ts_tablet_manager.cc:1367] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: Time spent bootstrapping tablet: real 0.058s	user 0.025s	sys 0.023s
I20240501 08:17:48.066794 25405 raft_consensus.cc:348] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } }
I20240501 08:17:48.066992 25405 raft_consensus.cc:729] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06c1439c9ab8415485eff20566811ede, State: Initialized, Role: LEARNER
I20240501 08:17:48.067217 25405 consensus_queue.cc:260] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: true } }
I20240501 08:17:48.067883 25405 ts_tablet_manager.cc:1398] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:17:48.068496 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session 06c1439c9ab8415485eff20566811ede-bfbef7af4781448bb04b107b53d58ca0 received from {username='slave'} at 127.2.248.132:41085
I20240501 08:17:48.068615 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session 06c1439c9ab8415485eff20566811ede-bfbef7af4781448bb04b107b53d58ca0 on tablet bfbef7af4781448bb04b107b53d58ca0 with peer 06c1439c9ab8415485eff20566811ede
W20240501 08:17:48.090698 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
W20240501 08:17:48.091241 10685 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.127396 10295 consensus_peers.cc:462] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:48.128216 25137 raft_consensus.cc:482] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c9f609481ded4d1aad9dd5f2341fd931)
I20240501 08:17:48.128346 25137 raft_consensus.cc:504] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:17:48.129092 25137 leader_election.cc:288] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749), c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153)
I20240501 08:17:48.129474 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
W20240501 08:17:48.130578 10558 leader_election.cc:334] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:17:48.130699 10558 leader_election.cc:302] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8d5f781e3ba94cbfbbea9b5782ef8eaf; no voters: 441e281fafbf42f8938c3f2f42ff7527, c3747ec2a93c4684962f215616e896ed
W20240501 08:17:48.130946 10557 leader_election.cc:334] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:48.130928 25137 raft_consensus.cc:2726] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:17:48.134764 10295 consensus_peers.cc:485] T d188d415f63f448195ae90e8e3d66e0e P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.151515 10425 consensus_peers.cc:485] T 8f27eb42e75247c2aa142d3028e6ca2a P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.154613 10295 consensus_peers.cc:485] T f5a58b7166534844b0963f4fab46dfd9 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.181577 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(03d0f89e1d8f40fe84d275b117d8145f): 4921 bytes on disk
I20240501 08:17:48.182185 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(03d0f89e1d8f40fe84d275b117d8145f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 08:17:48.183039 10798 maintenance_manager.cc:419] P 06c1439c9ab8415485eff20566811ede: Scheduling UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0): 4921 bytes on disk
I20240501 08:17:48.183468 10689 maintenance_manager.cc:643] P 06c1439c9ab8415485eff20566811ede: UndoDeltaBlockGCOp(bfbef7af4781448bb04b107b53d58ca0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 08:17:48.202567 25418 ts_tablet_manager.cc:902] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Initiating tablet copy from peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605)
I20240501 08:17:48.204082 25418 tablet_copy_client.cc:308] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46605
I20240501 08:17:48.204862 10512 tablet_copy_service.cc:140] P 861f629e2d134ff08f25fa553dfb5bda: Received BeginTabletCopySession request for tablet 185a62881ea84131a0c14987e4296338 from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf ({username='slave'} at 127.2.248.131:38611)
I20240501 08:17:48.205032 10512 tablet_copy_service.cc:161] P 861f629e2d134ff08f25fa553dfb5bda: Beginning new tablet copy session on tablet 185a62881ea84131a0c14987e4296338 from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf at {username='slave'} at 127.2.248.131:38611: session id = 8d5f781e3ba94cbfbbea9b5782ef8eaf-185a62881ea84131a0c14987e4296338
I20240501 08:17:48.207929 10512 tablet_copy_source_session.cc:205] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:48.209719 25418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185a62881ea84131a0c14987e4296338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:48.215310 25232 consensus_queue.cc:579] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.036s)
I20240501 08:17:48.215740 25418 tablet_copy_client.cc:783] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:48.219738 10749 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:17:48.222400 10619 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:48.222891 25418 tablet_copy_client.cc:647] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Starting download of 1 WAL segments...
W20240501 08:17:48.223026 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.223196 25423 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:17:48.224812 25418 tablet_copy_client.cc:523] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:48.225925 25232 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:48.226791 25220 raft_consensus.cc:482] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c9f609481ded4d1aad9dd5f2341fd931)
I20240501 08:17:48.226954 25220 raft_consensus.cc:504] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
I20240501 08:17:48.227691 25220 leader_election.cc:288] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319), 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749), c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153)
W20240501 08:17:48.229846 10297 leader_election.cc:334] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:48.229997 10298 leader_election.cc:334] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111)
I20240501 08:17:48.229879 10621 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:48.230125 10298 leader_election.cc:302] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c3747ec2a93c4684962f215616e896ed; no voters: 441e281fafbf42f8938c3f2f42ff7527, c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:17:48.230412 25220 raft_consensus.cc:2726] T bbef0147276d410da3d3830e682efe6d P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:17:48.231554 25418 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:17:48.233667 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21065b7dcc18456eb1b4c2376d51eec4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:48.234411 10221 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 9 to 10, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:48.234591 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbef0147276d410da3d3830e682efe6d" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" is_pre_election: true
I20240501 08:17:48.234740 10619 raft_consensus.cc:2454] T bbef0147276d410da3d3830e682efe6d P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 2.
I20240501 08:17:48.240085 10749 consensus_queue.cc:237] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:48.243515 10619 raft_consensus.cc:1261] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 08:17:48.244220 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21065b7dcc18456eb1b4c2376d51eec4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.244596 25232 consensus_queue.cc:1035] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:17:48.244957 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.246361 25232 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } } }
I20240501 08:17:48.249092 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 21065b7dcc18456eb1b4c2376d51eec4 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:48.250077 10619 raft_consensus.cc:2932] T 21065b7dcc18456eb1b4c2376d51eec4 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } } }
I20240501 08:17:48.261406 10221 catalog_manager.cc:5484] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 10 to 11, NON_VOTER 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130) added. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.262866 25418 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:48.263227 25418 tablet_bootstrap.cc:492] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap complete.
I20240501 08:17:48.263453 25418 ts_tablet_manager.cc:1367] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.032s	user 0.019s	sys 0.009s
I20240501 08:17:48.264515 25418 raft_consensus.cc:348] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
I20240501 08:17:48.264741 25418 raft_consensus.cc:729] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: LEARNER
I20240501 08:17:48.264914 25418 consensus_queue.cc:260] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
I20240501 08:17:48.265578 25418 ts_tablet_manager.cc:1398] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:17:48.267098 10512 tablet_copy_service.cc:342] P 861f629e2d134ff08f25fa553dfb5bda: Request end of tablet copy session 8d5f781e3ba94cbfbbea9b5782ef8eaf-185a62881ea84131a0c14987e4296338 received from {username='slave'} at 127.2.248.131:38611
I20240501 08:17:48.267199 10512 tablet_copy_service.cc:433] P 861f629e2d134ff08f25fa553dfb5bda: ending tablet copy session 8d5f781e3ba94cbfbbea9b5782ef8eaf-185a62881ea84131a0c14987e4296338 on tablet 185a62881ea84131a0c14987e4296338 with peer 8d5f781e3ba94cbfbbea9b5782ef8eaf
W20240501 08:17:48.276391 10428 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:17:48.311408 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:48.350019 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338): 4919 bytes on disk
I20240501 08:17:48.350509 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(185a62881ea84131a0c14987e4296338) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 08:17:48.377090 25050 consensus_queue.cc:579] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.092s)
I20240501 08:17:48.380847 10488 consensus_queue.cc:237] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }
W20240501 08:17:48.383577 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.383689 10358 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:17:48.384460 24218 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:17:48.386958 24218 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:17:48.388638 10358 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } }
I20240501 08:17:48.391346 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3453d8d883014219b1b14c88e250736b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:48.392374 10221 catalog_manager.cc:5484] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.397228 10488 consensus_queue.cc:237] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
I20240501 08:17:48.399266 10358 raft_consensus.cc:1261] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:48.399734 25050 consensus_queue.cc:1035] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:48.400470 24859 consensus_queue.cc:579] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.317s)
W20240501 08:17:48.401463 10425 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.402946 10428 consensus_peers.cc:485] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.403471 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:48.404009 24218 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.406980 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3453d8d883014219b1b14c88e250736b with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:48.408365 10358 consensus_queue.cc:237] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:17:48.408357 10362 raft_consensus.cc:2932] T 3453d8d883014219b1b14c88e250736b P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.413230 10221 catalog_manager.cc:5484] T 3453d8d883014219b1b14c88e250736b P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 8 to 9, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.413681 10488 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:17:48.414405 24859 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:17:48.415432 10295 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.420526 25383 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:48.421231 10488 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } }
I20240501 08:17:48.425714 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:48.426841 10221 catalog_manager.cc:5484] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 7 to 8, NON_VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.431969 10362 consensus_queue.cc:237] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
W20240501 08:17:48.433902 10295 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): Couldn't send request to peer 8d5f781e3ba94cbfbbea9b5782ef8eaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e56f5f3cd9f4a56a3d43543ac0e5d14. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:48.435209 10295 consensus_peers.cc:485] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.436574 10488 raft_consensus.cc:1261] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Refusing update from remote peer c3747ec2a93c4684962f215616e896ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:17:48.437261 24859 consensus_queue.cc:1035] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:17:48.440289 25271 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } } }
I20240501 08:17:48.443342 10207 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:48.446661 10221 catalog_manager.cc:5484] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed reported cstate change: config changed from index 8 to 9, NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) added. New cstate: current_term: 1 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.449168 10488 raft_consensus.cc:2932] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 861f629e2d134ff08f25fa553dfb5bda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } } }
I20240501 08:17:48.466922 10749 raft_consensus.cc:1203] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240501 08:17:48.481316 10749 raft_consensus.cc:1203] T 03d0f89e1d8f40fe84d275b117d8145f P 06c1439c9ab8415485eff20566811ede [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240501 08:17:48.488168 12719 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task3vHYYh/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240501 08:12:30.247845 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:30.248378 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:30.264298 12722 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:33.425063 12722 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:12:35.522064 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.522687 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:35.522854 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:35.522965 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.523159 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:35.523290 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:35.523413 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:35.523681 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:35.523775 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.523874 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc637c58564a476c8f56248349c3625e: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:35.524005 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:35.524145 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:35.524230 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:35.524361 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c4a9863b0a3c4aa0b5ab510cebd81045: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:35.524446 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3249ecbd03094e94a93017efa42b105c: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server bb2f0099fb794400b5c24927e213e5d8
I20240501 08:12:35.782248 12722 rebalancer_tool.cc:1074] tablet af24b8ab4d0c43ef986d89d4aefc263c: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:35.921675 12722 rebalancer_tool.cc:1084] tablet c665d7d20360440a9103e245f5f0900c: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move ignored: Illegal state: tablet c665d7d20360440a9103e245f5f0900c: replica bb2f0099fb794400b5c24927e213e5d8 is already present
I20240501 08:12:35.921847 12722 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:40.105909 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:40.106370 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62b3aca374c44c509e954361e2dcbb20: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.129366 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:40.129810 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.129935 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:40.130039 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 975dda02343941d7a896ed6459039ea6: could not find any suitable replica to move from server 441e281fafbf42f8938c3f2f42ff7527 to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:40.130261 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:40.130375 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e005ad4971044cda8ed96708209e8b34: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:40.130611 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:40.403887 12722 rebalancer_tool.cc:1074] tablet 850d4e1020e243f4b6fa113d2946846d: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:40.557147 12722 rebalancer_tool.cc:1084] tablet 8a6ed4ea6d6941d2af0f81eaf744bcea: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move ignored: Not found: Peer with uuid 8d5f781e3ba94cbfbbea9b5782ef8eaf not found in consensus config
I20240501 08:12:40.557271 12722 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:43.344488 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 8d5f781e3ba94cbfbbea9b5782ef8eaf to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:43.347365 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server 441e281fafbf42f8938c3f2f42ff7527
W20240501 08:12:43.347595 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 93d41bd6b8824bf68f8a23fecdbd5d89: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server bb2f0099fb794400b5c24927e213e5d8
W20240501 08:12:43.347702 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server 861f629e2d134ff08f25fa553dfb5bda to server b6a964e54f3b44eda6d0666e0c1e4bea
W20240501 08:12:43.347815 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6b8a9eb2e539416a8cfa53d67bb90b65: could not find any suitable replica to move from server c3747ec2a93c4684962f215616e896ed to server c9f609481ded4d1aad9dd5f2341fd931
W20240501 08:12:43.347956 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2a2bb2d036ee4460892b76fd8cda6ceb: could not find any suitable replica to move from server bb2f0099fb794400b5c24927e213e5d8 to server b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:12:43.666146 12722 rebalancer_tool.cc:1074] tablet 185a62881ea84131a0c14987e4296338: '06c1439c9ab8415485eff20566811ede' -> 'b6a964e54f3b44eda6d0666e0c1e4bea' move scheduled
I20240501 08:12:43.949770 12722 rebalancer_tool.cc:1074] tablet bbef0147276d410da3d3830e682efe6d: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move scheduled
I20240501 08:12:44.277118 12722 rebalancer_tool.cc:1084] tablet cd01dd912e1e4ca9a104270f3cd004b1: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> '' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } }
I20240501 08:12:44.277315 12722 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:46.267505 14884 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.2.248.130:46605, user_credentials={real_user=slave}} blocked reactor thread for 95242.1us
W20240501 08:12:46.980468 12722 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 169807e07dfe479c83d8666ceedc07a5: could not find any suitable replica to move from server 06c1439c9ab8415485eff20566811ede to server c9f609481ded4d1aad9dd5f2341fd931
I20240501 08:12:47.265662 12722 rebalancer_tool.cc:1074] tablet e99538651cf64510a811ba01c6c8fb40: 'c3747ec2a93c4684962f215616e896ed' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:47.975401 12722 rebalancer_tool.cc:1074] tablet 4b924c6344714a92ac37ae28961b02f5: '06c1439c9ab8415485eff20566811ede' -> '441e281fafbf42f8938c3f2f42ff7527' move scheduled
I20240501 08:12:48.108966 12722 rebalancer_tool.cc:1074] tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:48.219547 12722 rebalancer_tool.cc:1074] tablet 21065b7dcc18456eb1b4c2376d51eec4: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:48.349843 12722 rebalancer_tool.cc:1074] tablet 3453d8d883014219b1b14c88e250736b: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move scheduled
I20240501 08:12:48.745900 12722 rebalancer_tool.cc:1136] tablet 850d4e1020e243f4b6fa113d2946846d: '861f629e2d134ff08f25fa553dfb5bda' -> 'bb2f0099fb794400b5c24927e213e5d8' move completed: OK
I20240501 08:12:48.819720 12722 rebalancer_tool.cc:1136] tablet af24b8ab4d0c43ef986d89d4aefc263c: 'c3747ec2a93c4684962f215616e896ed' -> '441e281fafbf42f8938c3f2f42ff7527' move completed: Incomplete: tablet af24b8ab4d0c43ef986d89d4aefc263c, TS c3747ec2a93c4684962f215616e896ed -> TS 441e281fafbf42f8938c3f2f42ff7527 move failed, target replica disappeared
W20240501 08:17:47.281515 25305 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:17:47.205068 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.248.136:42153 (local address 127.0.0.1:59906)
0501 08:17:47.205476 (+   408us) negotiation.cc:107] Waiting for socket to connect
0501 08:17:47.205492 (+    16us) client_negotiation.cc:174] Beginning negotiation
0501 08:17:47.205605 (+   113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:17:47.278995 (+ 73390us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":202,"threads_started":1}
I20240501 08:17:47.282300 12722 rebalancer_tool.cc:1123] tablet bbef0147276d410da3d3830e682efe6d: '8d5f781e3ba94cbfbbea9b5782ef8eaf' -> 'c9f609481ded4d1aad9dd5f2341fd931' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:17:47.293768 12722 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): unacceptable health status UNAVAILABLE

E20240501 08:17:48.493024  3042 external_mini_cluster.cc:943] Process with UUID bb2f0099fb794400b5c24927e213e5d8 has crashed
E20240501 08:17:48.493129  3042 external_mini_cluster.cc:943] Process with UUID b6a964e54f3b44eda6d0666e0c1e4bea has crashed
E20240501 08:17:48.493188  3042 external_mini_cluster.cc:943] Process with UUID 441e281fafbf42f8938c3f2f42ff7527 has crashed
E20240501 08:17:48.493244  3042 external_mini_cluster.cc:943] Process with UUID c9f609481ded4d1aad9dd5f2341fd931 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 4
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240501 08:17:48.494513  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10280
I20240501 08:17:48.501084 10399 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:48.513931 10207 catalog_manager.cc:4573] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3453d8d883014219b1b14c88e250736b: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:17:48.609162 10619 raft_consensus.cc:1203] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20240501 08:17:48.687883 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:48.698424 25439 ts_tablet_manager.cc:902] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Initiating tablet copy from peer 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093)
I20240501 08:17:48.700023 25439 tablet_copy_client.cc:308] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:40093
I20240501 08:17:48.700639 10772 tablet_copy_service.cc:140] P 06c1439c9ab8415485eff20566811ede: Received BeginTabletCopySession request for tablet 21065b7dcc18456eb1b4c2376d51eec4 from peer 861f629e2d134ff08f25fa553dfb5bda ({username='slave'} at 127.2.248.130:34215)
I20240501 08:17:48.700789 10772 tablet_copy_service.cc:161] P 06c1439c9ab8415485eff20566811ede: Beginning new tablet copy session on tablet 21065b7dcc18456eb1b4c2376d51eec4 from peer 861f629e2d134ff08f25fa553dfb5bda at {username='slave'} at 127.2.248.130:34215: session id = 861f629e2d134ff08f25fa553dfb5bda-21065b7dcc18456eb1b4c2376d51eec4
I20240501 08:17:48.703307 10772 tablet_copy_source_session.cc:205] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:48.704546 25439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21065b7dcc18456eb1b4c2376d51eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:48.709208 25439 tablet_copy_client.cc:783] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:48.714463 25439 tablet_copy_client.cc:647] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:48.716030 25439 tablet_copy_client.cc:523] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:48.718361 25439 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap starting.
I20240501 08:17:48.735456 25439 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:48.735596 25050 consensus_queue.cc:579] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Leader has been unable to successfully communicate with peer 441e281fafbf42f8938c3f2f42ff7527 for more than 300 seconds (300.412s)
I20240501 08:17:48.735895 25439 tablet_bootstrap.cc:492] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Bootstrap complete.
I20240501 08:17:48.736202 25439 ts_tablet_manager.cc:1367] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Time spent bootstrapping tablet: real 0.018s	user 0.016s	sys 0.003s
I20240501 08:17:48.736969 25439 raft_consensus.cc:348] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:48.737251 25439 raft_consensus.cc:729] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 861f629e2d134ff08f25fa553dfb5bda, State: Initialized, Role: LEARNER
I20240501 08:17:48.737461 25439 consensus_queue.cc:260] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: NON_VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { promote: true } }
I20240501 08:17:48.737977 25439 ts_tablet_manager.cc:1398] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:17:48.739334 10488 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }
I20240501 08:17:48.741385 10772 tablet_copy_service.cc:342] P 06c1439c9ab8415485eff20566811ede: Request end of tablet copy session 861f629e2d134ff08f25fa553dfb5bda-21065b7dcc18456eb1b4c2376d51eec4 received from {username='slave'} at 127.2.248.130:34215
I20240501 08:17:48.741485 10772 tablet_copy_service.cc:433] P 06c1439c9ab8415485eff20566811ede: ending tablet copy session 861f629e2d134ff08f25fa553dfb5bda-21065b7dcc18456eb1b4c2376d51eec4 on tablet 21065b7dcc18456eb1b4c2376d51eec4 with peer 861f629e2d134ff08f25fa553dfb5bda
I20240501 08:17:48.741442 10619 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:17:48.742429 25432 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:17:48.744966 24218 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } }
I20240501 08:17:48.747570 10205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d4ab8e42a484cf89b42f5f53f511265 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:48.748025 10619 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } }
I20240501 08:17:48.750038 10218 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New cstate: current_term: 2 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:48.756409 10488 consensus_queue.cc:237] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:48.759609 10425 consensus_peers.cc:485] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.759653 10619 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:17:48.760416 24218 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:17:48.762265 25050 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.762774 10619 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.765810 10205 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6d4ab8e42a484cf89b42f5f53f511265 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:17:48.767361 10218 catalog_manager.cc:5484] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New cstate: current_term: 2 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:48.851586 25426 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b6a964e54f3b44eda6d0666e0c1e4bea)
I20240501 08:17:48.851771 25426 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:48.852450 25426 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049), b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
W20240501 08:17:48.853878 10685 leader_election.cc:334] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:48.868825 10358 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:17:48.870982 10362 raft_consensus.cc:1261] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:17:48.872606 24218 consensus_queue.cc:1035] T 6d4ab8e42a484cf89b42f5f53f511265 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:17:48.873643 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "06c1439c9ab8415485eff20566811ede" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "c3747ec2a93c4684962f215616e896ed" is_pre_election: true
I20240501 08:17:48.874177 10687 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06c1439c9ab8415485eff20566811ede; no voters: b6a964e54f3b44eda6d0666e0c1e4bea, c3747ec2a93c4684962f215616e896ed
I20240501 08:17:48.874527 25423 raft_consensus.cc:2726] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:17:48.875355 10358 raft_consensus.cc:858] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:48.875900 10358 raft_consensus.cc:1565] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Could not prepare op 'term: 2 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } }
W20240501 08:17:48.876729 10295 consensus_peers.cc:462] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:48.886068 10538 maintenance_manager.cc:419] P 861f629e2d134ff08f25fa553dfb5bda: Scheduling UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4): 4921 bytes on disk
I20240501 08:17:48.886888 10429 maintenance_manager.cc:643] P 861f629e2d134ff08f25fa553dfb5bda: UndoDeltaBlockGCOp(21065b7dcc18456eb1b4c2376d51eec4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240501 08:17:48.896705 10358 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } }
W20240501 08:17:48.902441 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:48.904560 10358 raft_consensus.cc:2932] T 6d4ab8e42a484cf89b42f5f53f511265 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: true } } }
I20240501 08:17:48.937062 25139 raft_consensus.cc:1013] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer 06c1439c9ab8415485eff20566811ede: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 08:17:48.943101 25453 ts_tablet_manager.cc:902] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Initiating tablet copy from peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049)
W20240501 08:17:48.949239 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
I20240501 08:17:48.963023 25453 tablet_copy_client.cc:308] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44049
I20240501 08:17:48.964155 10381 tablet_copy_service.cc:140] P c3747ec2a93c4684962f215616e896ed: Received BeginTabletCopySession request for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf ({username='slave'} at 127.2.248.131:44669)
I20240501 08:17:48.964358 10381 tablet_copy_service.cc:161] P c3747ec2a93c4684962f215616e896ed: Beginning new tablet copy session on tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 from peer 8d5f781e3ba94cbfbbea9b5782ef8eaf at {username='slave'} at 127.2.248.131:44669: session id = 8d5f781e3ba94cbfbbea9b5782ef8eaf-6e56f5f3cd9f4a56a3d43543ac0e5d14
I20240501 08:17:48.967959 10381 tablet_copy_source_session.cc:205] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P c3747ec2a93c4684962f215616e896ed: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:17:48.969630 25453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:48.978621 25453 tablet_copy_client.cc:783] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Starting download of 5 data blocks...
I20240501 08:17:48.984762 25453 tablet_copy_client.cc:647] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Starting download of 1 WAL segments...
I20240501 08:17:48.985970 25432 raft_consensus.cc:1050] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER 06c1439c9ab8415485eff20566811ede to VOTER
I20240501 08:17:48.986827 25453 tablet_copy_client.cc:523] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:17:48.987269 25432 consensus_queue.cc:237] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } }
I20240501 08:17:48.989881 25453 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap starting.
I20240501 08:17:48.990180 10749 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:48.990811 25050 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:17:48.992224 10427 consensus_peers.cc:485] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:48.993350 10619 raft_consensus.cc:1261] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:48.994503 25050 consensus_queue.cc:1035] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:17:48.998302 25050 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } } }
I20240501 08:17:48.999248 10619 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } } }
I20240501 08:17:49.007637 10749 raft_consensus.cc:2932] T bfbef7af4781448bb04b107b53d58ca0 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } } }
I20240501 08:17:49.009409 10218 catalog_manager.cc:5484] T bfbef7af4781448bb04b107b53d58ca0 P 861f629e2d134ff08f25fa553dfb5bda reported cstate change: config changed from index 9 to 10, 06c1439c9ab8415485eff20566811ede (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:49.010708 25426 consensus_queue.cc:579] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Leader has been unable to successfully communicate with peer bb2f0099fb794400b5c24927e213e5d8 for more than 300 seconds (300.084s)
I20240501 08:17:49.018540 10749 consensus_queue.cc:237] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } }
W20240501 08:17:49.030900 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:49.033902 10358 raft_consensus.cc:1261] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Refusing update from remote peer 06c1439c9ab8415485eff20566811ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:17:49.034960 25428 consensus_queue.cc:1035] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:17:49.039378 10688 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:49.040902 25428 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:49.045074 10207 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a6ed4ea6d6941d2af0f81eaf744bcea with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:17:49.045766 10358 raft_consensus.cc:2932] T 8a6ed4ea6d6941d2af0f81eaf744bcea P c3747ec2a93c4684962f215616e896ed [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } } }
I20240501 08:17:49.051034 10218 catalog_manager.cc:5484] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede reported cstate change: config changed from index 9 to 10, VOTER bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133) evicted. New cstate: current_term: 1 leader_uuid: "06c1439c9ab8415485eff20566811ede" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f609481ded4d1aad9dd5f2341fd931" member_type: VOTER last_known_addr { host: "127.2.248.136" port: 42153 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:49.054756 25453 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:17:49.055151 25453 tablet_bootstrap.cc:492] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Bootstrap complete.
I20240501 08:17:49.055433 25453 ts_tablet_manager.cc:1367] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent bootstrapping tablet: real 0.066s	user 0.038s	sys 0.019s
I20240501 08:17:49.056181 25453 raft_consensus.cc:348] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
I20240501 08:17:49.056895 25453 raft_consensus.cc:729] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d5f781e3ba94cbfbbea9b5782ef8eaf, State: Initialized, Role: LEARNER
I20240501 08:17:49.057463 25453 consensus_queue.cc:260] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: true } }
I20240501 08:17:49.058844 25453 ts_tablet_manager.cc:1398] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
W20240501 08:17:49.058497 10295 consensus_peers.cc:462] T 850d4e1020e243f4b6fa113d2946846d P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.060070 10381 tablet_copy_service.cc:342] P c3747ec2a93c4684962f215616e896ed: Request end of tablet copy session 8d5f781e3ba94cbfbbea9b5782ef8eaf-6e56f5f3cd9f4a56a3d43543ac0e5d14 received from {username='slave'} at 127.2.248.131:44669
I20240501 08:17:49.060232 10381 tablet_copy_service.cc:433] P c3747ec2a93c4684962f215616e896ed: ending tablet copy session 8d5f781e3ba94cbfbbea9b5782ef8eaf-6e56f5f3cd9f4a56a3d43543ac0e5d14 on tablet 6e56f5f3cd9f4a56a3d43543ac0e5d14 with peer 8d5f781e3ba94cbfbbea9b5782ef8eaf
I20240501 08:17:49.104575 10668 maintenance_manager.cc:419] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: Scheduling UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14): 4581 bytes on disk
I20240501 08:17:49.105216 10559 maintenance_manager.cc:643] P 8d5f781e3ba94cbfbbea9b5782ef8eaf: UndoDeltaBlockGCOp(6e56f5f3cd9f4a56a3d43543ac0e5d14) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 08:17:49.130558 10492 raft_consensus.cc:1203] T 21065b7dcc18456eb1b4c2376d51eec4 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240501 08:17:49.140324 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
W20240501 08:17:49.157265 10557 consensus_peers.cc:485] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.201092 25050 raft_consensus.cc:1050] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda: attempting to promote NON_VOTER 8d5f781e3ba94cbfbbea9b5782ef8eaf to VOTER
I20240501 08:17:49.201757 25050 consensus_queue.cc:237] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } }
I20240501 08:17:49.204108 10749 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:17:49.205085 10619 raft_consensus.cc:1261] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Refusing update from remote peer 861f629e2d134ff08f25fa553dfb5bda: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:17:49.205075 25050 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240501 08:17:49.206233 25050 consensus_queue.cc:1035] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:17:49.206502 10425 consensus_peers.cc:485] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:49.210173 10688 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer 441e281fafbf42f8938c3f2f42ff7527 (127.2.248.135:37749): Couldn't send request to peer 441e281fafbf42f8938c3f2f42ff7527. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.135:37749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.210642 24218 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 861f629e2d134ff08f25fa553dfb5bda [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.211548 10619 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.214277 10749 raft_consensus.cc:2932] T 185a62881ea84131a0c14987e4296338 P 06c1439c9ab8415485eff20566811ede [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.219579 25447 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b6a964e54f3b44eda6d0666e0c1e4bea)
I20240501 08:17:49.219843 25447 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:49.221019 25447 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
I20240501 08:17:49.223078 10218 catalog_manager.cc:5484] T 185a62881ea84131a0c14987e4296338 P 8d5f781e3ba94cbfbbea9b5782ef8eaf reported cstate change: config changed from index 10 to 11, 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "861f629e2d134ff08f25fa553dfb5bda" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "861f629e2d134ff08f25fa553dfb5bda" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46605 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "8d5f781e3ba94cbfbbea9b5782ef8eaf" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34319 } attrs { promote: false } } }
I20240501 08:17:49.224290 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede" is_pre_election: true
I20240501 08:17:49.224568 10749 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 2.
W20240501 08:17:49.224606 10295 leader_election.cc:334] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.226182 10297 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:17:49.228401 25447 raft_consensus.cc:2781] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:17:49.228519 25447 raft_consensus.cc:482] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Starting leader election (detected failure of leader b6a964e54f3b44eda6d0666e0c1e4bea)
I20240501 08:17:49.228653 25447 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:17:49.231871 25447 raft_consensus.cc:504] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:49.233088 25447 leader_election.cc:288] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 election: Requested vote from peers 06c1439c9ab8415485eff20566811ede (127.2.248.132:40093), b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821)
W20240501 08:17:49.234838 10295 leader_election.cc:334] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.234009 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd01dd912e1e4ca9a104270f3cd004b1" candidate_uuid: "c3747ec2a93c4684962f215616e896ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "06c1439c9ab8415485eff20566811ede"
I20240501 08:17:49.235810 10749 raft_consensus.cc:3037] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:17:49.240084 10749 raft_consensus.cc:2454] T cd01dd912e1e4ca9a104270f3cd004b1 P 06c1439c9ab8415485eff20566811ede [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3747ec2a93c4684962f215616e896ed in term 3.
I20240501 08:17:49.240626 10297 leader_election.cc:302] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06c1439c9ab8415485eff20566811ede, c3747ec2a93c4684962f215616e896ed; no voters: b6a964e54f3b44eda6d0666e0c1e4bea
I20240501 08:17:49.240967 25447 raft_consensus.cc:2781] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:17:49.241154 25447 raft_consensus.cc:686] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [term 3 LEADER]: Becoming Leader. State: Replica: c3747ec2a93c4684962f215616e896ed, State: Running, Role: LEADER
I20240501 08:17:49.241535 25447 consensus_queue.cc:237] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } }
I20240501 08:17:49.250631 10221 catalog_manager.cc:5484] T cd01dd912e1e4ca9a104270f3cd004b1 P c3747ec2a93c4684962f215616e896ed reported cstate change: term changed from 2 to 3, leader changed from b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134) to c3747ec2a93c4684962f215616e896ed (127.2.248.129). New cstate: current_term: 3 leader_uuid: "c3747ec2a93c4684962f215616e896ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3747ec2a93c4684962f215616e896ed" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1439c9ab8415485eff20566811ede" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 40093 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a964e54f3b44eda6d0666e0c1e4bea" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 44821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441e281fafbf42f8938c3f2f42ff7527" member_type: NON_VOTER last_known_addr { host: "127.2.248.135" port: 37749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:17:49.326258 10619 raft_consensus.cc:1203] T 6e56f5f3cd9f4a56a3d43543ac0e5d14 P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240501 08:17:49.357326 25137 raft_consensus.cc:1013] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer 06c1439c9ab8415485eff20566811ede: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 08:17:49.377364 10295 consensus_peers.cc:462] T ae946c165cd34853917e5c8a244b6408 P c3747ec2a93c4684962f215616e896ed -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
I20240501 08:17:49.399592 10210 maintenance_manager.cc:419] P ee59fa3365af4b91907d4ab87092780e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10424 bytes on disk
I20240501 08:17:49.403685 10209 maintenance_manager.cc:643] P ee59fa3365af4b91907d4ab87092780e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s	user 0.004s	sys 0.000s Metrics: {"thread_start_us":231,"threads_started":1}
W20240501 08:17:49.482102 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:49.497063  3042 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10280: Timed out: CheckLeaks RPC to 127.2.248.129:44049 timed out after 1.000s (SENT)
I20240501 08:17:49.497287  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10280
W20240501 08:17:49.516992 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:49.518471 10557 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.557607  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10410
W20240501 08:17:49.562269 10207 catalog_manager.cc:4573] TS bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3453d8d883014219b1b14c88e250736b: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111)
I20240501 08:17:49.565677 10532 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:49.649559 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.681677 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.706872 10687 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.749640 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
W20240501 08:17:49.793797 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:17:49.828708 25137 raft_consensus.cc:1013] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf [term 1 LEADER]: attempt to promote peer 06c1439c9ab8415485eff20566811ede: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 08:17:49.894850 10687 connection.cc:535] client connection to 127.2.248.129:44049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:49.896448 10687 consensus_peers.cc:485] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:49.896579 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:49.933763 10687 consensus_peers.cc:485] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:49.936336 10557 consensus_peers.cc:485] T 84789f1228f64be5b6976a9d52f64bd0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:49.941272  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10410
W20240501 08:17:49.954964 10557 consensus_peers.cc:485] T 03d0f89e1d8f40fe84d275b117d8145f P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:50.002514 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
LLVM ERROR: IO failure on output stream: Broken pipe
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcdfce)[0x55b53b8e6fce]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc27d)[0x55b53b8e527d]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc3d3)[0x55b53b8e53d3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f057d628980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f057c706fb7]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f057c708921]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa575c)[0x55b53b8be75c]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa58e0)[0x55b53b8be8e0]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xbcc73)[0x55b53b8d5c73]
/lib/x86_64-linux-gnu/libc.so.6(+0x43161)[0x7f057c70b161]
/lib/x86_64-linux-gnu/libc.so.6(+0x4325a)[0x7f057c70b25a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xee)[0x7f057c6e9bfe]
/tmp/dist-test-task3vHYYh/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0x8f0ba)[0x55b53b8a80ba]
W20240501 08:17:50.021636 10687 connection.cc:535] server connection from 127.2.248.130:34215 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:50.021683 10685 connection.cc:535] client connection to 127.2.248.130:46605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:17:50.021867 10555 connection.cc:535] client connection to 127.2.248.130:46605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:17:50.022436  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10540
W20240501 08:17:50.023203 10555 consensus_peers.cc:485] T 9b6f09b0b5e84329944360d511578ed0 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.023204 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.023329 10685 consensus_peers.cc:485] T 2a63e16f11894c2886091dcd3d204106 P 06c1439c9ab8415485eff20566811ede -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.023334 10555 consensus_peers.cc:485] T aa6139020ac7458aa8a7ac08f63e6f26 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer 861f629e2d134ff08f25fa553dfb5bda (127.2.248.130:46605): Couldn't send request to peer 861f629e2d134ff08f25fa553dfb5bda. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:50.029034 10662 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:50.033936 10687 consensus_peers.cc:485] T 8a6ed4ea6d6941d2af0f81eaf744bcea P 06c1439c9ab8415485eff20566811ede -> Peer c3747ec2a93c4684962f215616e896ed (127.2.248.129:44049): Couldn't send request to peer c3747ec2a93c4684962f215616e896ed. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:17:50.139729 10687 consensus_peers.cc:462] T b72323b8b99f4fd1aedcce69089f6204 P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:50.178439 10685 consensus_peers.cc:462] T 50b7150bbeb645d1b00d1a561c3276d1 P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111)
W20240501 08:17:50.256088 10687 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Couldn't send request to peer c9f609481ded4d1aad9dd5f2341fd931. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:50.405244 10557 consensus_peers.cc:485] T 6624441a1f1143648913595f905f8039 P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer bb2f0099fb794400b5c24927e213e5d8 (127.2.248.133:43981): Couldn't send request to peer bb2f0099fb794400b5c24927e213e5d8. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.133:43981: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:17:50.406117  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10540
W20240501 08:17:50.406148 10557 consensus_peers.cc:462] T ea502ae00eff4f72a74b8d25f5a11ede P 8d5f781e3ba94cbfbbea9b5782ef8eaf -> Peer c9f609481ded4d1aad9dd5f2341fd931 (127.2.248.136:42153): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.2.248.136:42153: connect: Connection refused (error 111)
W20240501 08:17:50.457290 10685 consensus_peers.cc:485] T af24b8ab4d0c43ef986d89d4aefc263c P 06c1439c9ab8415485eff20566811ede -> Peer b6a964e54f3b44eda6d0666e0c1e4bea (127.2.248.134:44821): Couldn't send request to peer b6a964e54f3b44eda6d0666e0c1e4bea. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.134:44821: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:17:50.461288 10685 connection.cc:535] client connection to 127.2.248.131:34319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:17:50.461697  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10670
W20240501 08:17:50.462452 10685 consensus_peers.cc:485] T 21065b7dcc18456eb1b4c2376d51eec4 P 06c1439c9ab8415485eff20566811ede -> Peer 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): Couldn't send request to peer 8d5f781e3ba94cbfbbea9b5782ef8eaf. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:17:50.467391 10789 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:17:50.508368 10205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34319: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 08:17:50.564241 10205 catalog_manager.cc:4573] TS 8d5f781e3ba94cbfbbea9b5782ef8eaf (127.2.248.131:34319): ChangeConfig:ADD_PEER:NON_VOTER RPC failed for tablet 03d0f89e1d8f40fe84d275b117d8145f: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34319: connect: Connection refused (error 111)
I20240501 08:17:50.793329  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10670
I20240501 08:17:50.846485  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 10190
I20240501 08:17:50.851821 10249 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:17:51.103031  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 10190
2024-05-01T08:17:51Z chronyd exiting
I20240501 08:17:51.169014  3042 test_util.cc:175] -----------------------------------------------
I20240501 08:17:51.169097  3042 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714551038306655-3042-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (346228 ms)
[----------] 1 test from ConcurrentRebalancersTest (346228 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240501 08:17:51.171156  3042 test_util.cc:268] Using random seed: 526607936
I20240501 08:17:51.172892  3042 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:17:51.172967  3042 ts_itest-base.cc:113] --------------
I20240501 08:17:51.173002  3042 ts_itest-base.cc:114] 4 tablet servers
I20240501 08:17:51.173040  3042 ts_itest-base.cc:115] 1 replicas per TS
I20240501 08:17:51.173079  3042 ts_itest-base.cc:116] --------------
2024-05-01T08:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:17:51Z Disabled control of system clock
I20240501 08:17:51.213475  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:45945
--webserver_interface=127.2.248.190
--webserver_port=0
--builtin_ntp_servers=127.2.248.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.248.190:45945
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240501 08:17:51.549934 25505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:17:51.550228 25505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:17:51.550318 25505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:17:51.557577 25505 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:17:51.557695 25505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:17:51.557754 25505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:17:51.557821 25505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:17:51.557917 25505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:17:51.566596 25505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.2.248.190:45945
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:45945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.248.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:17:51.567859 25505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:17:51.569608 25505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:17:51.580245 25513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:51.580319 25510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:51.580749 25511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:17:51.581382 25505 server_base.cc:1034] running on GCE node
I20240501 08:17:51.582100 25505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:17:51.583412 25505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:17:51.584592 25505 hybrid_clock.cc:648] HybridClock initialized: now 1714551471584546 us; error 38 us; skew 500 ppm
I20240501 08:17:51.584921 25505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:17:51.591353 25505 webserver.cc:460] Webserver started at http://127.2.248.190:39103/ using document root <none> and password file <none>
I20240501 08:17:51.592108 25505 fs_manager.cc:359] Metadata directory not provided
I20240501 08:17:51.592253 25505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:17:51.592568 25505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:17:51.595057 25505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "04ebf9d3a1754802ad27defb32c04321"
format_stamp: "Formatted at 2024-05-01 08:17:51 on dist-test-slave-0pdg"
I20240501 08:17:51.595791 25505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "04ebf9d3a1754802ad27defb32c04321"
format_stamp: "Formatted at 2024-05-01 08:17:51 on dist-test-slave-0pdg"
I20240501 08:17:51.600683 25505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240501 08:17:51.603789 25519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:51.605099 25505 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:17:51.605247 25505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
uuid: "04ebf9d3a1754802ad27defb32c04321"
format_stamp: "Formatted at 2024-05-01 08:17:51 on dist-test-slave-0pdg"
I20240501 08:17:51.605427 25505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:17:51.624771 25505 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:17:51.625540 25505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:17:51.625806 25505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:17:51.642395 25505 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.190:45945
I20240501 08:17:51.642401 25570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.190:45945 every 8 connection(s)
I20240501 08:17:51.643666 25505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:17:51.647200 25571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:51.649605  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 25505
I20240501 08:17:51.649883  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:17:51.657948 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321: Bootstrap starting.
I20240501 08:17:51.661020 25571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:51.662206 25571 log.cc:824] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321: Log is configured to *not* fsync() on all Append() calls
I20240501 08:17:51.664834 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321: No bootstrap required, opened a new log
I20240501 08:17:51.670827 25571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ebf9d3a1754802ad27defb32c04321" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45945 } }
I20240501 08:17:51.671149 25571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:51.671218 25571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ebf9d3a1754802ad27defb32c04321, State: Initialized, Role: FOLLOWER
I20240501 08:17:51.671761 25571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ebf9d3a1754802ad27defb32c04321" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45945 } }
I20240501 08:17:51.671931 25571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:51.672020 25571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:51.672159 25571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:51.674219 25571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ebf9d3a1754802ad27defb32c04321" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45945 } }
I20240501 08:17:51.674634 25571 leader_election.cc:302] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ebf9d3a1754802ad27defb32c04321; no voters: 
I20240501 08:17:51.675159 25571 leader_election.cc:288] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:51.675293 25576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:51.675891 25576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [term 1 LEADER]: Becoming Leader. State: Replica: 04ebf9d3a1754802ad27defb32c04321, State: Running, Role: LEADER
I20240501 08:17:51.676401 25576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ebf9d3a1754802ad27defb32c04321" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45945 } }
I20240501 08:17:51.676998 25571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:17:51.679296 25578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ebf9d3a1754802ad27defb32c04321. Latest consensus state: current_term: 1 leader_uuid: "04ebf9d3a1754802ad27defb32c04321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ebf9d3a1754802ad27defb32c04321" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45945 } } }
I20240501 08:17:51.679296 25577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04ebf9d3a1754802ad27defb32c04321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ebf9d3a1754802ad27defb32c04321" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45945 } } }
I20240501 08:17:51.679693 25578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [sys.catalog]: This master's current role is: LEADER
I20240501 08:17:51.679746 25577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321 [sys.catalog]: This master's current role is: LEADER
I20240501 08:17:51.681268 25583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:17:51.687579 25583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:17:51.698024 25583 catalog_manager.cc:1347] Generated new cluster ID: 068dd829713d464e9ee18d0b054f56bb
I20240501 08:17:51.698159 25583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:17:51.705960 25583 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:17:51.707578 25583 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:17:51.719328 25583 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 04ebf9d3a1754802ad27defb32c04321: Generated new TSK 0
I20240501 08:17:51.720218 25583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:17:51.728847  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.129:0
--local_ip_for_outbound_sockets=127.2.248.129
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--builtin_ntp_servers=127.2.248.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:17:52.072983 25595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:17:52.073262 25595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:17:52.073387 25595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:17:52.081329 25595 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:17:52.081442 25595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:17:52.081602 25595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.129
I20240501 08:17:52.090481 25595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.129
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:17:52.091727 25595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:17:52.093381 25595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:17:52.104558 25600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:52.105016 25601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:17:52.105556 25595 server_base.cc:1034] running on GCE node
W20240501 08:17:52.105944 25603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:17:52.106537 25595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:17:52.107386 25595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:17:52.108575 25595 hybrid_clock.cc:648] HybridClock initialized: now 1714551472108532 us; error 31 us; skew 500 ppm
I20240501 08:17:52.108871 25595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:17:52.110975 25595 webserver.cc:460] Webserver started at http://127.2.248.129:36307/ using document root <none> and password file <none>
I20240501 08:17:52.111656 25595 fs_manager.cc:359] Metadata directory not provided
I20240501 08:17:52.111757 25595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:17:52.112036 25595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:17:52.114518 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
format_stamp: "Formatted at 2024-05-01 08:17:52 on dist-test-slave-0pdg"
I20240501 08:17:52.115221 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
format_stamp: "Formatted at 2024-05-01 08:17:52 on dist-test-slave-0pdg"
I20240501 08:17:52.120342 25595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20240501 08:17:52.123602 25609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:52.124840 25595 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:17:52.124976 25595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
format_stamp: "Formatted at 2024-05-01 08:17:52 on dist-test-slave-0pdg"
I20240501 08:17:52.125154 25595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:17:52.158663 25595 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:17:52.159483 25595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:17:52.159809 25595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:17:52.160797 25595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:17:52.162240 25595 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:17:52.162322 25595 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:52.162422 25595 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:17:52.162503 25595 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:52.197350 25595 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.129:40347
I20240501 08:17:52.197405 25721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.129:40347 every 8 connection(s)
I20240501 08:17:52.198566 25595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:17:52.201481  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 25595
I20240501 08:17:52.201656  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:17:52.204461  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.130:0
--local_ip_for_outbound_sockets=127.2.248.130
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--builtin_ntp_servers=127.2.248.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:17:52.210736 25722 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45945
I20240501 08:17:52.211068 25722 heartbeater.cc:464] Registering TS with master...
I20240501 08:17:52.212008 25722 heartbeater.cc:511] Master 127.2.248.190:45945 requested a full tablet report, sending...
I20240501 08:17:52.214464 25536 ts_manager.cc:194] Registered new tserver with Master: 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:17:52.215907 25536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.129:37005
W20240501 08:17:52.532236 25726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:17:52.532522 25726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:17:52.532658 25726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:17:52.540196 25726 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:17:52.540331 25726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:17:52.540517 25726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.130
I20240501 08:17:52.549412 25726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.130
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:17:52.550698 25726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:17:52.552354 25726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:17:52.563413 25731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:52.563900 25734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:52.564302 25732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:17:52.564911 25726 server_base.cc:1034] running on GCE node
I20240501 08:17:52.565445 25726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:17:52.566339 25726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:17:52.567521 25726 hybrid_clock.cc:648] HybridClock initialized: now 1714551472567472 us; error 43 us; skew 500 ppm
I20240501 08:17:52.567796 25726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:17:52.569808 25726 webserver.cc:460] Webserver started at http://127.2.248.130:39631/ using document root <none> and password file <none>
I20240501 08:17:52.570457 25726 fs_manager.cc:359] Metadata directory not provided
I20240501 08:17:52.570561 25726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:17:52.570845 25726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:17:52.573351 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a839e332a8bd470ab44d040cd50c81e6"
format_stamp: "Formatted at 2024-05-01 08:17:52 on dist-test-slave-0pdg"
I20240501 08:17:52.574075 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a839e332a8bd470ab44d040cd50c81e6"
format_stamp: "Formatted at 2024-05-01 08:17:52 on dist-test-slave-0pdg"
I20240501 08:17:52.578840 25726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 08:17:52.581728 25740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:52.582825 25726 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:17:52.582954 25726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a839e332a8bd470ab44d040cd50c81e6"
format_stamp: "Formatted at 2024-05-01 08:17:52 on dist-test-slave-0pdg"
I20240501 08:17:52.583117 25726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:17:52.605197 25726 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:17:52.606007 25726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:17:52.606251 25726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:17:52.607182 25726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:17:52.608673 25726 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:17:52.608772 25726 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:52.608871 25726 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:17:52.608935 25726 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:52.639374 25726 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.130:45313
I20240501 08:17:52.639407 25852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.130:45313 every 8 connection(s)
I20240501 08:17:52.640671 25726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:17:52.644690  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 25726
I20240501 08:17:52.644869  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:17:52.648401  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:0
--local_ip_for_outbound_sockets=127.2.248.131
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--builtin_ntp_servers=127.2.248.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:17:52.655186 25853 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45945
I20240501 08:17:52.655498 25853 heartbeater.cc:464] Registering TS with master...
I20240501 08:17:52.656524 25853 heartbeater.cc:511] Master 127.2.248.190:45945 requested a full tablet report, sending...
I20240501 08:17:52.658124 25536 ts_manager.cc:194] Registered new tserver with Master: a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130:45313)
I20240501 08:17:52.659071 25536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.130:47777
W20240501 08:17:52.977819 25856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:17:52.978106 25856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:17:52.978221 25856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:17:52.985939 25856 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:17:52.986073 25856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:17:52.986241 25856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:17:52.995296 25856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:17:52.996552 25856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:17:52.998217 25856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:17:53.009608 25861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:53.010067 25862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:53.010566 25864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:17:53.011250 25856 server_base.cc:1034] running on GCE node
I20240501 08:17:53.011773 25856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:17:53.012636 25856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:17:53.013841 25856 hybrid_clock.cc:648] HybridClock initialized: now 1714551473013794 us; error 45 us; skew 500 ppm
I20240501 08:17:53.014137 25856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:17:53.016165 25856 webserver.cc:460] Webserver started at http://127.2.248.131:36291/ using document root <none> and password file <none>
I20240501 08:17:53.016888 25856 fs_manager.cc:359] Metadata directory not provided
I20240501 08:17:53.016988 25856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:17:53.017302 25856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:17:53.019713 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "946da6d9a36a48b9b67c494ccc6df85e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:17:53.020443 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "946da6d9a36a48b9b67c494ccc6df85e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:17:53.025112 25856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 08:17:53.028436 25870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:53.029842 25856 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:17:53.029992 25856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "946da6d9a36a48b9b67c494ccc6df85e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:17:53.030172 25856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:17:53.051713 25856 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:17:53.052593 25856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:17:53.052865 25856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:17:53.053779 25856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:17:53.055259 25856 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:17:53.055356 25856 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:53.055473 25856 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:17:53.055553 25856 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:53.086707 25856 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:45885
I20240501 08:17:53.086774 25982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:45885 every 8 connection(s)
I20240501 08:17:53.088027 25856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:17:53.088502  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 25856
I20240501 08:17:53.088716  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:17:53.092427  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:0
--local_ip_for_outbound_sockets=127.2.248.132
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--builtin_ntp_servers=127.2.248.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:17:53.102499 25983 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45945
I20240501 08:17:53.102818 25983 heartbeater.cc:464] Registering TS with master...
I20240501 08:17:53.103600 25983 heartbeater.cc:511] Master 127.2.248.190:45945 requested a full tablet report, sending...
I20240501 08:17:53.105062 25536 ts_manager.cc:194] Registered new tserver with Master: 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131:45885)
I20240501 08:17:53.106189 25536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:47359
I20240501 08:17:53.218660 25722 heartbeater.cc:503] Master 127.2.248.190:45945 was elected leader, sending a full tablet report...
W20240501 08:17:53.418726 25986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:17:53.419020 25986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:17:53.419145 25986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:17:53.426409 25986 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:17:53.426545 25986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:17:53.426702 25986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:17:53.435343 25986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:17:53.436583 25986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:17:53.438212 25986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:17:53.449205 25991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:53.449685 25992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:17:53.450198 25994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:17:53.450457 25986 server_base.cc:1034] running on GCE node
I20240501 08:17:53.451027 25986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:17:53.451944 25986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:17:53.453181 25986 hybrid_clock.cc:648] HybridClock initialized: now 1714551473453134 us; error 37 us; skew 500 ppm
I20240501 08:17:53.453491 25986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:17:53.456357 25986 webserver.cc:460] Webserver started at http://127.2.248.132:37517/ using document root <none> and password file <none>
I20240501 08:17:53.457103 25986 fs_manager.cc:359] Metadata directory not provided
I20240501 08:17:53.457213 25986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:17:53.457501 25986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:17:53.460026 25986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6a94029ae37f45e39162bb542ad7a97e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:17:53.460777 25986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6a94029ae37f45e39162bb542ad7a97e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:17:53.465530 25986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 08:17:53.471397 26000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:53.472816 25986 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:17:53.472975 25986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6a94029ae37f45e39162bb542ad7a97e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:17:53.473183 25986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:17:53.497661 25986 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:17:53.498459 25986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:17:53.498708 25986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:17:53.499580 25986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:17:53.500960 25986 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:17:53.501155 25986 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:53.501274 25986 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:17:53.501345 25986 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:17:53.531725 25986 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:35633
I20240501 08:17:53.531780 26112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:35633 every 8 connection(s)
I20240501 08:17:53.533053 25986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:17:53.543185  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 25986
I20240501 08:17:53.543383  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:17:53.544560 26113 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45945
I20240501 08:17:53.544826 26113 heartbeater.cc:464] Registering TS with master...
I20240501 08:17:53.545662 26113 heartbeater.cc:511] Master 127.2.248.190:45945 requested a full tablet report, sending...
I20240501 08:17:53.546836 25536 ts_manager.cc:194] Registered new tserver with Master: 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132:35633)
I20240501 08:17:53.547597 25536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:47527
I20240501 08:17:53.558442  3042 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 08:17:53.574014 25536 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47722:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240501 08:17:53.600135 25657 tablet_service.cc:1467] Processing CreateTablet for tablet 5394e26d870c48969f2d8670c188691a (DEFAULT_TABLE table=TestTable [id=112434fea13441cf93ed191eddc48203]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:53.601300 25657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5394e26d870c48969f2d8670c188691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:53.610873 26127 tablet_bootstrap.cc:492] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Bootstrap starting.
I20240501 08:17:53.614135 26127 tablet_bootstrap.cc:654] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:53.615134 26127 log.cc:824] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Log is configured to *not* fsync() on all Append() calls
I20240501 08:17:53.617419 26127 tablet_bootstrap.cc:492] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: No bootstrap required, opened a new log
I20240501 08:17:53.617653 26127 ts_tablet_manager.cc:1367] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.005s
I20240501 08:17:53.623811 26127 raft_consensus.cc:348] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:53.624138 26127 raft_consensus.cc:374] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:53.624241 26127 raft_consensus.cc:729] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Initialized, Role: FOLLOWER
I20240501 08:17:53.624800 26127 consensus_queue.cc:260] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:53.624985 26127 raft_consensus.cc:388] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:53.625097 26127 raft_consensus.cc:482] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:53.625234 26127 raft_consensus.cc:3037] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:53.627043 26127 raft_consensus.cc:504] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:53.627453 26127 leader_election.cc:302] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd; no voters: 
I20240501 08:17:53.628015 26127 leader_election.cc:288] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:53.628178 26129 raft_consensus.cc:2781] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:53.629544 26129 raft_consensus.cc:686] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Leader. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:17:53.629892 26127 ts_tablet_manager.cc:1398] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20240501 08:17:53.630136 26129 consensus_queue.cc:237] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:53.635411 25536 catalog_manager.cc:5484] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129). New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:53.645568  3042 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 08:17:53.647555  3042 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4bbaf16fd6f6495ea9ba00f007ad03cd to finish bootstrapping
I20240501 08:17:53.657159  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a839e332a8bd470ab44d040cd50c81e6 to finish bootstrapping
I20240501 08:17:53.661455 25853 heartbeater.cc:503] Master 127.2.248.190:45945 was elected leader, sending a full tablet report...
I20240501 08:17:53.665988  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 946da6d9a36a48b9b67c494ccc6df85e to finish bootstrapping
I20240501 08:17:53.674760  3042 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6a94029ae37f45e39162bb542ad7a97e to finish bootstrapping
I20240501 08:17:53.683262  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 25856
I20240501 08:17:53.684175 25978 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:17:53.761332  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 25856
I20240501 08:17:53.778857  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 25986
I20240501 08:17:53.780061 26108 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:17:53.859141  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 25986
I20240501 08:17:57.627421 25535 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47722:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:17:57.637557 25657 tablet_service.cc:1467] Processing CreateTablet for tablet 108a1ac12fa04ce9a525a20d1680f4bd (DEFAULT_TABLE table=rebalance_test_table_0 [id=8836558475de402991d6ab54c997ba6b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:57.637955 25657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108a1ac12fa04ce9a525a20d1680f4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:57.645035 26146 tablet_bootstrap.cc:492] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Bootstrap starting.
I20240501 08:17:57.647374 26146 tablet_bootstrap.cc:654] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:57.646836 25787 tablet_service.cc:1467] Processing CreateTablet for tablet 64562a17576d48e7b8e2b079d24ec1b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8836558475de402991d6ab54c997ba6b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:57.647542 25788 tablet_service.cc:1467] Processing CreateTablet for tablet d81f2b5e8ee6478888520c203baef7ee (DEFAULT_TABLE table=rebalance_test_table_0 [id=8836558475de402991d6ab54c997ba6b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:57.648713 25788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f2b5e8ee6478888520c203baef7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:57.650854 25787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64562a17576d48e7b8e2b079d24ec1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:57.657310 26146 tablet_bootstrap.cc:492] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd: No bootstrap required, opened a new log
I20240501 08:17:57.657485 26146 ts_tablet_manager.cc:1367] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20240501 08:17:57.658349 26146 raft_consensus.cc:348] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:57.658546 26146 raft_consensus.cc:374] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:57.658661 26146 raft_consensus.cc:729] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Initialized, Role: FOLLOWER
I20240501 08:17:57.658926 26146 consensus_queue.cc:260] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:57.659138 26146 raft_consensus.cc:388] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:57.659227 26146 raft_consensus.cc:482] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:57.659368 26146 raft_consensus.cc:3037] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:57.662047 26146 raft_consensus.cc:504] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:57.662380 26146 leader_election.cc:302] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd; no voters: 
I20240501 08:17:57.663393 26146 leader_election.cc:288] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:57.663511 26149 raft_consensus.cc:2781] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:57.663877 26148 tablet_bootstrap.cc:492] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6: Bootstrap starting.
I20240501 08:17:57.664409 26149 raft_consensus.cc:686] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Leader. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:17:57.664934 26149 consensus_queue.cc:237] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:57.667179 25535 catalog_manager.cc:5484] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129). New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:57.667491 26148 tablet_bootstrap.cc:654] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:57.668315 26146 ts_tablet_manager.cc:1398] T 108a1ac12fa04ce9a525a20d1680f4bd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.003s
I20240501 08:17:57.669217 26148 log.cc:824] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6: Log is configured to *not* fsync() on all Append() calls
I20240501 08:17:57.677924 26148 tablet_bootstrap.cc:492] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6: No bootstrap required, opened a new log
I20240501 08:17:57.678160 26148 ts_tablet_manager.cc:1367] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.008s
I20240501 08:17:57.683760 26148 raft_consensus.cc:348] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.684056 26148 raft_consensus.cc:374] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:57.684185 26148 raft_consensus.cc:729] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Initialized, Role: FOLLOWER
I20240501 08:17:57.684742 26148 consensus_queue.cc:260] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.684928 26148 raft_consensus.cc:388] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:57.685053 26148 raft_consensus.cc:482] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:57.685189 26148 raft_consensus.cc:3037] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:57.687091 26148 raft_consensus.cc:504] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.687553 26148 leader_election.cc:302] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:17:57.688254 26148 leader_election.cc:288] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:57.688362 26155 raft_consensus.cc:2781] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:57.688943 26155 raft_consensus.cc:686] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Leader. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:17:57.689819 26155 consensus_queue.cc:237] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.690011 26148 ts_tablet_manager.cc:1398] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.004s
I20240501 08:17:57.690354 26148 tablet_bootstrap.cc:492] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: Bootstrap starting.
I20240501 08:17:57.691972 26148 tablet_bootstrap.cc:654] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:57.694368 26148 tablet_bootstrap.cc:492] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: No bootstrap required, opened a new log
I20240501 08:17:57.694299 25536 catalog_manager.cc:5484] T 64562a17576d48e7b8e2b079d24ec1b4 P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:57.694509 26148 ts_tablet_manager.cc:1367] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:17:57.695140 26148 raft_consensus.cc:348] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.695297 26148 raft_consensus.cc:374] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:57.695389 26148 raft_consensus.cc:729] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Initialized, Role: FOLLOWER
I20240501 08:17:57.695596 26148 consensus_queue.cc:260] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.695775 26148 raft_consensus.cc:388] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:57.695858 26148 raft_consensus.cc:482] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:57.695952 26148 raft_consensus.cc:3037] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:57.697955 26148 raft_consensus.cc:504] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.698148 26148 leader_election.cc:302] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:17:57.698310 26148 leader_election.cc:288] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:57.698402 26155 raft_consensus.cc:2781] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:57.698549 26155 raft_consensus.cc:686] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Leader. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:17:57.698767 26148 ts_tablet_manager.cc:1398] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 08:17:57.698783 26155 consensus_queue.cc:237] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:57.700739 25536 catalog_manager.cc:5484] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:59.069970 25536 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47722:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:17:59.078107 25657 tablet_service.cc:1467] Processing CreateTablet for tablet a09ad135fef749a1a292070a9a2fdbaa (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7d7552aef85466e9309288251370a5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:59.078302 25787 tablet_service.cc:1467] Processing CreateTablet for tablet 2c511b81062a47b388bc886786bffca0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7d7552aef85466e9309288251370a5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:59.078565 25657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09ad135fef749a1a292070a9a2fdbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:59.078683 25787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c511b81062a47b388bc886786bffca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:59.078506 25656 tablet_service.cc:1467] Processing CreateTablet for tablet f14ed0c6ed914ce79e58ecb219b159dd (DEFAULT_TABLE table=rebalance_test_table_1 [id=d7d7552aef85466e9309288251370a5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:59.079001 25656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14ed0c6ed914ce79e58ecb219b159dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:59.083900 26187 tablet_bootstrap.cc:492] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Bootstrap starting.
I20240501 08:17:59.085063 26188 tablet_bootstrap.cc:492] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6: Bootstrap starting.
I20240501 08:17:59.085744 26187 tablet_bootstrap.cc:654] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:59.086692 26188 tablet_bootstrap.cc:654] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:59.087728 26187 tablet_bootstrap.cc:492] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: No bootstrap required, opened a new log
I20240501 08:17:59.087941 26187 ts_tablet_manager.cc:1367] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:17:59.088654 26188 tablet_bootstrap.cc:492] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6: No bootstrap required, opened a new log
I20240501 08:17:59.088804 26188 ts_tablet_manager.cc:1367] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:17:59.088718 26187 raft_consensus.cc:348] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.088886 26187 raft_consensus.cc:374] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:59.088986 26187 raft_consensus.cc:729] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Initialized, Role: FOLLOWER
I20240501 08:17:59.089218 26187 consensus_queue.cc:260] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.089409 26187 raft_consensus.cc:388] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:59.089495 26187 raft_consensus.cc:482] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:59.089614 26187 raft_consensus.cc:3037] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:59.089555 26188 raft_consensus.cc:348] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.089708 26188 raft_consensus.cc:374] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:59.089815 26188 raft_consensus.cc:729] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Initialized, Role: FOLLOWER
I20240501 08:17:59.090052 26188 consensus_queue.cc:260] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.090236 26188 raft_consensus.cc:388] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:59.090312 26188 raft_consensus.cc:482] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:59.090411 26188 raft_consensus.cc:3037] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:59.092271 26187 raft_consensus.cc:504] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.092507 26187 leader_election.cc:302] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd; no voters: 
I20240501 08:17:59.092914 26187 leader_election.cc:288] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:59.093060 26191 raft_consensus.cc:2781] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:59.093003 26188 raft_consensus.cc:504] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.093271 26188 leader_election.cc:302] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:17:59.096411 26188 leader_election.cc:288] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:59.096673 26194 raft_consensus.cc:2781] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:59.099539 26191 raft_consensus.cc:686] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Leader. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:17:59.100378 26194 raft_consensus.cc:686] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Leader. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:17:59.100759 26194 consensus_queue.cc:237] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.100844 26187 ts_tablet_manager.cc:1398] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.011s
I20240501 08:17:59.100526 26191 consensus_queue.cc:237] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.101306 26187 tablet_bootstrap.cc:492] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd: Bootstrap starting.
I20240501 08:17:59.102330 26188 ts_tablet_manager.cc:1398] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.004s
I20240501 08:17:59.103312 25536 catalog_manager.cc:5484] T 2c511b81062a47b388bc886786bffca0 P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:59.103860 26187 tablet_bootstrap.cc:654] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:59.106300 25535 catalog_manager.cc:5484] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129). New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:59.109727 26187 tablet_bootstrap.cc:492] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd: No bootstrap required, opened a new log
I20240501 08:17:59.109921 26187 ts_tablet_manager.cc:1367] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240501 08:17:59.110605 26187 raft_consensus.cc:348] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.110736 26187 raft_consensus.cc:374] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:59.110787 26187 raft_consensus.cc:729] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Initialized, Role: FOLLOWER
I20240501 08:17:59.110977 26187 consensus_queue.cc:260] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.111090 26187 raft_consensus.cc:388] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:59.111153 26187 raft_consensus.cc:482] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:59.111212 26187 raft_consensus.cc:3037] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:59.114136 26187 raft_consensus.cc:504] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.114338 26187 leader_election.cc:302] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd; no voters: 
I20240501 08:17:59.114544 26187 leader_election.cc:288] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:59.114593 26191 raft_consensus.cc:2781] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:59.114821 26191 raft_consensus.cc:686] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Leader. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:17:59.115074 26187 ts_tablet_manager.cc:1398] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240501 08:17:59.115053 26191 consensus_queue.cc:237] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.116909 25535 catalog_manager.cc:5484] T a09ad135fef749a1a292070a9a2fdbaa P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129). New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:59.593299 25535 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47722:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:17:59.600243 25787 tablet_service.cc:1467] Processing CreateTablet for tablet 12b5316170e144648629fa542cc3d49c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1459d36a672c447e9b6cb0bfbb3869e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:59.600380 25788 tablet_service.cc:1467] Processing CreateTablet for tablet b619c0c69bf644c4aff333388d10ea4e (DEFAULT_TABLE table=rebalance_test_table_2 [id=1459d36a672c447e9b6cb0bfbb3869e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:59.600718 25787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b5316170e144648629fa542cc3d49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:59.600684 25657 tablet_service.cc:1467] Processing CreateTablet for tablet 9db572dece424334958f71a54c91c779 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1459d36a672c447e9b6cb0bfbb3869e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:17:59.600948 25788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b619c0c69bf644c4aff333388d10ea4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:59.601080 25657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db572dece424334958f71a54c91c779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:17:59.607223 26187 tablet_bootstrap.cc:492] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Bootstrap starting.
I20240501 08:17:59.607349 26224 tablet_bootstrap.cc:492] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: Bootstrap starting.
I20240501 08:17:59.608847 26187 tablet_bootstrap.cc:654] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:59.608892 26224 tablet_bootstrap.cc:654] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:59.610706 26187 tablet_bootstrap.cc:492] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: No bootstrap required, opened a new log
I20240501 08:17:59.610821 26187 ts_tablet_manager.cc:1367] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 08:17:59.611348 26187 raft_consensus.cc:348] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.611474 26187 raft_consensus.cc:374] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:59.611537 26187 raft_consensus.cc:729] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Initialized, Role: FOLLOWER
I20240501 08:17:59.611692 26187 consensus_queue.cc:260] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.611824 26187 raft_consensus.cc:388] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:59.611874 26187 raft_consensus.cc:482] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:59.611944 26187 raft_consensus.cc:3037] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:59.612586 26224 tablet_bootstrap.cc:492] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: No bootstrap required, opened a new log
I20240501 08:17:59.612708 26224 ts_tablet_manager.cc:1367] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 08:17:59.613303 26224 raft_consensus.cc:348] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.613432 26224 raft_consensus.cc:374] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:59.613523 26224 raft_consensus.cc:729] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Initialized, Role: FOLLOWER
I20240501 08:17:59.613660 26187 raft_consensus.cc:504] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.613754 26224 consensus_queue.cc:260] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.613921 26187 leader_election.cc:302] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd; no voters: 
I20240501 08:17:59.613960 26224 raft_consensus.cc:388] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:59.614060 26224 raft_consensus.cc:482] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:59.614118 26187 leader_election.cc:288] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:59.614177 26224 raft_consensus.cc:3037] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:59.614187 26191 raft_consensus.cc:2781] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:59.614320 26191 raft_consensus.cc:686] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Leader. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:17:59.614545 26191 consensus_queue.cc:237] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:17:59.615542 26187 ts_tablet_manager.cc:1398] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:17:59.616248 26224 raft_consensus.cc:504] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.616451 26224 leader_election.cc:302] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:17:59.616547 25535 catalog_manager.cc:5484] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129). New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:59.616889 26224 leader_election.cc:288] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:59.617017 26228 raft_consensus.cc:2781] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:59.617645 26224 ts_tablet_manager.cc:1398] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.004s
I20240501 08:17:59.617722 26228 raft_consensus.cc:686] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Leader. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:17:59.617903 26224 tablet_bootstrap.cc:492] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6: Bootstrap starting.
I20240501 08:17:59.617987 26228 consensus_queue.cc:237] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.619377 26224 tablet_bootstrap.cc:654] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6: Neither blocks nor log segments found. Creating new log.
I20240501 08:17:59.621748 25535 catalog_manager.cc:5484] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } health_report { overall_health: HEALTHY } } }
I20240501 08:17:59.627856 26224 tablet_bootstrap.cc:492] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6: No bootstrap required, opened a new log
I20240501 08:17:59.628024 26224 ts_tablet_manager.cc:1367] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240501 08:17:59.628954 26224 raft_consensus.cc:348] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.629160 26224 raft_consensus.cc:374] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:17:59.629266 26224 raft_consensus.cc:729] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Initialized, Role: FOLLOWER
I20240501 08:17:59.629510 26224 consensus_queue.cc:260] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.629629 26224 raft_consensus.cc:388] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:17:59.629699 26224 raft_consensus.cc:482] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:17:59.629765 26224 raft_consensus.cc:3037] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:17:59.631376 26224 raft_consensus.cc:504] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.631527 26224 leader_election.cc:302] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:17:59.631676 26224 leader_election.cc:288] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:17:59.631744 26228 raft_consensus.cc:2781] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:17:59.631893 26228 raft_consensus.cc:686] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Leader. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:17:59.632145 26224 ts_tablet_manager.cc:1398] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240501 08:17:59.632089 26228 consensus_queue.cc:237] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:17:59.633849 25535 catalog_manager.cc:5484] T b619c0c69bf644c4aff333388d10ea4e P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:00.103514 25535 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47722:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240501 08:18:00.110505 25657 tablet_service.cc:1467] Processing CreateTablet for tablet 706f5a79f2f34a21831cdcd9ac6b0aab (DEFAULT_TABLE table=rebalance_test_table_3 [id=c2b5627bcd7546e793340948e2c12233]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:00.110690 25787 tablet_service.cc:1467] Processing CreateTablet for tablet 78a27dc2d0d1404a916493386059b301 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c2b5627bcd7546e793340948e2c12233]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:00.110982 25657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706f5a79f2f34a21831cdcd9ac6b0aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:00.111143 25787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a27dc2d0d1404a916493386059b301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:00.111222 25656 tablet_service.cc:1467] Processing CreateTablet for tablet a7c9814a54714f0d99d2d2a16e46ea61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c2b5627bcd7546e793340948e2c12233]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:00.111526 25656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c9814a54714f0d99d2d2a16e46ea61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:00.115541 26224 tablet_bootstrap.cc:492] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6: Bootstrap starting.
I20240501 08:18:00.117139 26224 tablet_bootstrap.cc:654] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:00.118436 26255 tablet_bootstrap.cc:492] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Bootstrap starting.
I20240501 08:18:00.118809 26224 tablet_bootstrap.cc:492] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6: No bootstrap required, opened a new log
I20240501 08:18:00.118947 26224 ts_tablet_manager.cc:1367] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:00.119491 26224 raft_consensus.cc:348] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:18:00.119566 26255 tablet_bootstrap.cc:654] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:00.119623 26224 raft_consensus.cc:374] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:00.119716 26224 raft_consensus.cc:729] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Initialized, Role: FOLLOWER
I20240501 08:18:00.119916 26224 consensus_queue.cc:260] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:18:00.120069 26224 raft_consensus.cc:388] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:18:00.120172 26224 raft_consensus.cc:482] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:18:00.120270 26224 raft_consensus.cc:3037] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:00.121101 26255 tablet_bootstrap.cc:492] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: No bootstrap required, opened a new log
I20240501 08:18:00.121204 26255 ts_tablet_manager.cc:1367] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:00.121737 26255 raft_consensus.cc:348] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.121853 26255 raft_consensus.cc:374] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:00.121920 26255 raft_consensus.cc:729] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Initialized, Role: FOLLOWER
I20240501 08:18:00.122128 26255 consensus_queue.cc:260] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.122262 26255 raft_consensus.cc:388] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:18:00.122316 26255 raft_consensus.cc:482] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:18:00.122399 26255 raft_consensus.cc:3037] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:00.122314 26224 raft_consensus.cc:504] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:18:00.122809 26224 leader_election.cc:302] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:18:00.123040 26224 leader_election.cc:288] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:18:00.123131 26228 raft_consensus.cc:2781] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:00.123272 26228 raft_consensus.cc:686] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Leader. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:18:00.123523 26224 ts_tablet_manager.cc:1398] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:18:00.123486 26228 consensus_queue.cc:237] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } }
I20240501 08:18:00.124591 26255 raft_consensus.cc:504] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.124830 26255 leader_election.cc:302] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd; no voters: 
I20240501 08:18:00.125278 26255 leader_election.cc:288] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:18:00.125502 26259 raft_consensus.cc:2781] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:00.126008 26259 raft_consensus.cc:686] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Leader. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:18:00.126304 26259 consensus_queue.cc:237] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.126427 25535 catalog_manager.cc:5484] T 78a27dc2d0d1404a916493386059b301 P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130). New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:00.127511 26255 ts_tablet_manager.cc:1398] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240501 08:18:00.127796 26255 tablet_bootstrap.cc:492] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd: Bootstrap starting.
I20240501 08:18:00.128672 25536 catalog_manager.cc:5484] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129). New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:00.129288 26255 tablet_bootstrap.cc:654] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:00.132076 26255 tablet_bootstrap.cc:492] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd: No bootstrap required, opened a new log
I20240501 08:18:00.132248 26255 ts_tablet_manager.cc:1367] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:18:00.132936 26255 raft_consensus.cc:348] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.133101 26255 raft_consensus.cc:374] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:00.133188 26255 raft_consensus.cc:729] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Initialized, Role: FOLLOWER
I20240501 08:18:00.133380 26255 consensus_queue.cc:260] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.133492 26255 raft_consensus.cc:388] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:18:00.133577 26255 raft_consensus.cc:482] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:18:00.133643 26255 raft_consensus.cc:3037] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:00.135691 26255 raft_consensus.cc:504] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.135885 26255 leader_election.cc:302] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd; no voters: 
I20240501 08:18:00.136068 26255 leader_election.cc:288] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:18:00.136143 26260 raft_consensus.cc:2781] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:00.136262 26260 raft_consensus.cc:686] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Leader. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:18:00.136538 26255 ts_tablet_manager.cc:1398] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240501 08:18:00.136533 26260 consensus_queue.cc:237] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } }
I20240501 08:18:00.138553 25536 catalog_manager.cc:5484] T 706f5a79f2f34a21831cdcd9ac6b0aab P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129). New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:00.628578  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:45885
--local_ip_for_outbound_sockets=127.2.248.131
--tserver_master_addrs=127.2.248.190:45945
--webserver_port=36291
--webserver_interface=127.2.248.131
--builtin_ntp_servers=127.2.248.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:18:00.945756 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:00.946024 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:00.946108 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:00.952967 26287 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:00.953065 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:00.953166 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:18:00.961297 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:45885
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=36291
--tserver_master_addrs=127.2.248.190:45945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:00.962497 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:00.964056 26287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:00.975216 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:00.975605 26295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:00.976143 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:00.976708 26287 server_base.cc:1034] running on GCE node
I20240501 08:18:00.977295 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:00.978127 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:00.979336 26287 hybrid_clock.cc:648] HybridClock initialized: now 1714551480979301 us; error 30 us; skew 500 ppm
I20240501 08:18:00.979635 26287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:00.981649 26287 webserver.cc:460] Webserver started at http://127.2.248.131:36291/ using document root <none> and password file <none>
I20240501 08:18:00.982312 26287 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:00.982429 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:00.987254 26287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.002s
I20240501 08:18:00.989918 26301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:00.991122 26287 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:18:00.991277 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "946da6d9a36a48b9b67c494ccc6df85e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:18:00.992024 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:01.006676 26287 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:01.007385 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:01.007643 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:01.008558 26287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:01.010003 26287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:01.010076 26287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:01.010200 26287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:01.010253 26287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:01.040998 26287 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:45885
I20240501 08:18:01.041035 26413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:45885 every 8 connection(s)
I20240501 08:18:01.042477 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:18:01.044101  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 26287
I20240501 08:18:01.044924  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:35633
--local_ip_for_outbound_sockets=127.2.248.132
--tserver_master_addrs=127.2.248.190:45945
--webserver_port=37517
--webserver_interface=127.2.248.132
--builtin_ntp_servers=127.2.248.148:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:01.057307 26414 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45945
I20240501 08:18:01.057574 26414 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:01.058344 26414 heartbeater.cc:511] Master 127.2.248.190:45945 requested a full tablet report, sending...
I20240501 08:18:01.059916 25536 ts_manager.cc:194] Re-registered known tserver with Master: 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131:45885)
I20240501 08:18:01.061036 25536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:34955
W20240501 08:18:01.376168 26418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:01.376461 26418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:01.376658 26418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:01.384016 26418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:01.384167 26418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:01.384332 26418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:18:01.393445 26418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:35633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=37517
--tserver_master_addrs=127.2.248.190:45945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:01.394651 26418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:01.396368 26418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:01.407845 26426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:01.408353 26424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:01.408787 26423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:01.409444 26418 server_base.cc:1034] running on GCE node
I20240501 08:18:01.410035 26418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:01.410935 26418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:01.412176 26418 hybrid_clock.cc:648] HybridClock initialized: now 1714551481412107 us; error 55 us; skew 500 ppm
I20240501 08:18:01.412479 26418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:01.414520 26418 webserver.cc:460] Webserver started at http://127.2.248.132:37517/ using document root <none> and password file <none>
I20240501 08:18:01.415191 26418 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:01.415310 26418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:01.420107 26418 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:01.422866 26432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:01.424059 26418 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:01.424242 26418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6a94029ae37f45e39162bb542ad7a97e"
format_stamp: "Formatted at 2024-05-01 08:17:53 on dist-test-slave-0pdg"
I20240501 08:18:01.424965 26418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:01.440619 26418 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:01.441331 26418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:01.441569 26418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:01.442345 26418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:01.443595 26418 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:01.443666 26418 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:01.443789 26418 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:01.443840 26418 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:01.473033 26418 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:35633
I20240501 08:18:01.473068 26544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:35633 every 8 connection(s)
I20240501 08:18:01.474300 26418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:18:01.476233  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 26418
I20240501 08:18:01.493595 26545 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45945
I20240501 08:18:01.493913 26545 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:01.494827 26545 heartbeater.cc:511] Master 127.2.248.190:45945 requested a full tablet report, sending...
I20240501 08:18:01.496296 25536 ts_manager.cc:194] Re-registered known tserver with Master: 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132:35633)
I20240501 08:18:01.497059 25536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:43237
I20240501 08:18:01.923630 25656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:01.933219 26480 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:01.937351 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:01.945382 25787 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:02.063550 26414 heartbeater.cc:503] Master 127.2.248.190:45945 was elected leader, sending a full tablet report...
I20240501 08:18:02.098022 25656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.106009 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:02.106549 25787 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:02.117753 26480 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:02.259090 25656 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.261085 26480 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:02.272891 26349 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:02.275038 25787 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:02.341408 25808 consensus_queue.cc:237] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:02.343766 26661 raft_consensus.cc:2932] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } } }
I20240501 08:18:02.349282 25534 catalog_manager.cc:5484] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: config changed from index -1 to 3, NON_VOTER 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:02.353893 25741 consensus_peers.cc:485] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 -> Peer 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132:35633): Couldn't send request to peer 6a94029ae37f45e39162bb542ad7a97e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12b5316170e144648629fa542cc3d49c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:02.360949 25677 consensus_queue.cc:237] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.363523 26674 raft_consensus.cc:2932] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } } }
I20240501 08:18:02.368716 25534 catalog_manager.cc:5484] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index -1 to 2, NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) added. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:02.371574 25613 consensus_peers.cc:485] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd -> Peer 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131:45885): Couldn't send request to peer 946da6d9a36a48b9b67c494ccc6df85e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14ed0c6ed914ce79e58ecb219b159dd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:02.383713 25808 consensus_queue.cc:237] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.385223 26661 raft_consensus.cc:2932] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } } }
I20240501 08:18:02.390054 25534 catalog_manager.cc:5484] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: config changed from index -1 to 3, NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) added. New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:02.391803 25744 consensus_peers.cc:485] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 -> Peer 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131:45885): Couldn't send request to peer 946da6d9a36a48b9b67c494ccc6df85e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81f2b5e8ee6478888520c203baef7ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:02.400307 25677 consensus_queue.cc:237] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:02.401770 26674 raft_consensus.cc:2932] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } } }
I20240501 08:18:02.406625 25534 catalog_manager.cc:5484] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index -1 to 2, NON_VOTER 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:02.408586 25610 consensus_peers.cc:485] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd -> Peer 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132:35633): Couldn't send request to peer 6a94029ae37f45e39162bb542ad7a97e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7c9814a54714f0d99d2d2a16e46ea61. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:02.417433 25677 consensus_queue.cc:237] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.418864 26673 raft_consensus.cc:2932] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } } }
W20240501 08:18:02.419236 25613 consensus_peers.cc:485] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd -> Peer 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131:45885): Couldn't send request to peer 946da6d9a36a48b9b67c494ccc6df85e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9db572dece424334958f71a54c91c779. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:02.423141 25534 catalog_manager.cc:5484] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index -1 to 2, NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) added. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:02.434862 25677 consensus_queue.cc:237] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:02.436033 26673 raft_consensus.cc:2932] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } } }
W20240501 08:18:02.436488 25610 consensus_peers.cc:485] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd -> Peer 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132:35633): Couldn't send request to peer 6a94029ae37f45e39162bb542ad7a97e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5394e26d870c48969f2d8670c188691a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:02.439688 25534 catalog_manager.cc:5484] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index -1 to 2, NON_VOTER 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:02.499804 26545 heartbeater.cc:503] Master 127.2.248.190:45945 was elected leader, sending a full tablet report...
I20240501 08:18:02.779072 26761 ts_tablet_manager.cc:902] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: Initiating tablet copy from peer a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130:45313)
I20240501 08:18:02.780858 26761 tablet_copy_client.cc:308] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45313
I20240501 08:18:02.786586 25828 tablet_copy_service.cc:140] P a839e332a8bd470ab44d040cd50c81e6: Received BeginTabletCopySession request for tablet 12b5316170e144648629fa542cc3d49c from peer 6a94029ae37f45e39162bb542ad7a97e ({username='slave'} at 127.2.248.132:59147)
I20240501 08:18:02.786847 25828 tablet_copy_service.cc:161] P a839e332a8bd470ab44d040cd50c81e6: Beginning new tablet copy session on tablet 12b5316170e144648629fa542cc3d49c from peer 6a94029ae37f45e39162bb542ad7a97e at {username='slave'} at 127.2.248.132:59147: session id = 6a94029ae37f45e39162bb542ad7a97e-12b5316170e144648629fa542cc3d49c
I20240501 08:18:02.789428 25828 tablet_copy_source_session.cc:205] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:02.793529 26761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b5316170e144648629fa542cc3d49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:02.802378 26761 tablet_copy_client.cc:783] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:02.802650 26761 tablet_copy_client.cc:647] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:02.804466 26761 tablet_copy_client.cc:523] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:02.807052 26761 tablet_bootstrap.cc:492] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap starting.
I20240501 08:18:02.810545 26767 ts_tablet_manager.cc:902] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: Initiating tablet copy from peer a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130:45313)
I20240501 08:18:02.812422 26767 tablet_copy_client.cc:308] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:45313
I20240501 08:18:02.813784 26761 log.cc:824] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:02.819093 25828 tablet_copy_service.cc:140] P a839e332a8bd470ab44d040cd50c81e6: Received BeginTabletCopySession request for tablet d81f2b5e8ee6478888520c203baef7ee from peer 946da6d9a36a48b9b67c494ccc6df85e ({username='slave'} at 127.2.248.131:33925)
I20240501 08:18:02.819250 25828 tablet_copy_service.cc:161] P a839e332a8bd470ab44d040cd50c81e6: Beginning new tablet copy session on tablet d81f2b5e8ee6478888520c203baef7ee from peer 946da6d9a36a48b9b67c494ccc6df85e at {username='slave'} at 127.2.248.131:33925: session id = 946da6d9a36a48b9b67c494ccc6df85e-d81f2b5e8ee6478888520c203baef7ee
I20240501 08:18:02.821874 25828 tablet_copy_source_session.cc:205] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:02.823079 26761 tablet_bootstrap.cc:492] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:02.823424 26761 tablet_bootstrap.cc:492] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap complete.
I20240501 08:18:02.823737 26761 ts_tablet_manager.cc:1367] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: Time spent bootstrapping tablet: real 0.017s	user 0.015s	sys 0.000s
I20240501 08:18:02.825413 26767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f2b5e8ee6478888520c203baef7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:02.827582 26761 raft_consensus.cc:348] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:02.827894 26761 raft_consensus.cc:729] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a94029ae37f45e39162bb542ad7a97e, State: Initialized, Role: LEARNER
I20240501 08:18:02.828321 26761 consensus_queue.cc:260] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:02.829813 26761 ts_tablet_manager.cc:1398] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e: Time spent starting tablet: real 0.006s	user 0.009s	sys 0.000s
I20240501 08:18:02.830693 25828 tablet_copy_service.cc:342] P a839e332a8bd470ab44d040cd50c81e6: Request end of tablet copy session 6a94029ae37f45e39162bb542ad7a97e-12b5316170e144648629fa542cc3d49c received from {username='slave'} at 127.2.248.132:59147
I20240501 08:18:02.830833 25828 tablet_copy_service.cc:433] P a839e332a8bd470ab44d040cd50c81e6: ending tablet copy session 6a94029ae37f45e39162bb542ad7a97e-12b5316170e144648629fa542cc3d49c on tablet 12b5316170e144648629fa542cc3d49c with peer 6a94029ae37f45e39162bb542ad7a97e
I20240501 08:18:02.836143 26767 tablet_copy_client.cc:783] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:02.836437 26767 tablet_copy_client.cc:647] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:02.838143 26767 tablet_copy_client.cc:523] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:02.841046 26767 tablet_bootstrap.cc:492] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap starting.
I20240501 08:18:02.846913 26767 log.cc:824] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:02.854127 26767 tablet_bootstrap.cc:492] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:02.854436 26767 tablet_bootstrap.cc:492] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap complete.
I20240501 08:18:02.854725 26767 ts_tablet_manager.cc:1367] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.000s
I20240501 08:18:02.858630 26767 raft_consensus.cc:348] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.858928 26767 raft_consensus.cc:729] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 946da6d9a36a48b9b67c494ccc6df85e, State: Initialized, Role: LEARNER
I20240501 08:18:02.859265 26767 consensus_queue.cc:260] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.860448 26767 ts_tablet_manager.cc:1398] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.001s
I20240501 08:18:02.861220 25828 tablet_copy_service.cc:342] P a839e332a8bd470ab44d040cd50c81e6: Request end of tablet copy session 946da6d9a36a48b9b67c494ccc6df85e-d81f2b5e8ee6478888520c203baef7ee received from {username='slave'} at 127.2.248.131:33925
I20240501 08:18:02.861325 25828 tablet_copy_service.cc:433] P a839e332a8bd470ab44d040cd50c81e6: ending tablet copy session 946da6d9a36a48b9b67c494ccc6df85e-d81f2b5e8ee6478888520c203baef7ee on tablet d81f2b5e8ee6478888520c203baef7ee with peer 946da6d9a36a48b9b67c494ccc6df85e
I20240501 08:18:02.883284 26767 ts_tablet_manager.cc:902] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: Initiating tablet copy from peer 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:02.884502 26767 tablet_copy_client.cc:308] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:40347
I20240501 08:18:02.889714 25697 tablet_copy_service.cc:140] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Received BeginTabletCopySession request for tablet 9db572dece424334958f71a54c91c779 from peer 946da6d9a36a48b9b67c494ccc6df85e ({username='slave'} at 127.2.248.131:58505)
I20240501 08:18:02.889959 25697 tablet_copy_service.cc:161] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Beginning new tablet copy session on tablet 9db572dece424334958f71a54c91c779 from peer 946da6d9a36a48b9b67c494ccc6df85e at {username='slave'} at 127.2.248.131:58505: session id = 946da6d9a36a48b9b67c494ccc6df85e-9db572dece424334958f71a54c91c779
I20240501 08:18:02.892490 25697 tablet_copy_source_session.cc:205] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:02.893816 26767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db572dece424334958f71a54c91c779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:02.897905 26767 tablet_copy_client.cc:783] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:02.898121 26767 tablet_copy_client.cc:647] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:02.899601 26767 tablet_copy_client.cc:523] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:02.901628 26767 tablet_bootstrap.cc:492] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap starting.
I20240501 08:18:02.905776 26767 tablet_bootstrap.cc:492] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:02.906109 26767 tablet_bootstrap.cc:492] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap complete.
I20240501 08:18:02.906296 26767 ts_tablet_manager.cc:1367] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.004s
I20240501 08:18:02.906993 26767 raft_consensus.cc:348] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.907158 26767 raft_consensus.cc:729] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 946da6d9a36a48b9b67c494ccc6df85e, State: Initialized, Role: LEARNER
I20240501 08:18:02.907354 26767 consensus_queue.cc:260] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.907812 26767 ts_tablet_manager.cc:1398] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:18:02.908644 25697 tablet_copy_service.cc:342] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Request end of tablet copy session 946da6d9a36a48b9b67c494ccc6df85e-9db572dece424334958f71a54c91c779 received from {username='slave'} at 127.2.248.131:58505
I20240501 08:18:02.908816 25697 tablet_copy_service.cc:433] P 4bbaf16fd6f6495ea9ba00f007ad03cd: ending tablet copy session 946da6d9a36a48b9b67c494ccc6df85e-9db572dece424334958f71a54c91c779 on tablet 9db572dece424334958f71a54c91c779 with peer 946da6d9a36a48b9b67c494ccc6df85e
I20240501 08:18:02.929872 26767 ts_tablet_manager.cc:902] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: Initiating tablet copy from peer 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:02.930944 26767 tablet_copy_client.cc:308] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:40347
I20240501 08:18:02.931502 25697 tablet_copy_service.cc:140] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Received BeginTabletCopySession request for tablet f14ed0c6ed914ce79e58ecb219b159dd from peer 946da6d9a36a48b9b67c494ccc6df85e ({username='slave'} at 127.2.248.131:58505)
I20240501 08:18:02.931638 25697 tablet_copy_service.cc:161] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Beginning new tablet copy session on tablet f14ed0c6ed914ce79e58ecb219b159dd from peer 946da6d9a36a48b9b67c494ccc6df85e at {username='slave'} at 127.2.248.131:58505: session id = 946da6d9a36a48b9b67c494ccc6df85e-f14ed0c6ed914ce79e58ecb219b159dd
I20240501 08:18:02.933354 25697 tablet_copy_source_session.cc:205] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:02.934269 26767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14ed0c6ed914ce79e58ecb219b159dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:02.937956 26767 tablet_copy_client.cc:783] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:02.938167 26767 tablet_copy_client.cc:647] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:02.939369 26767 tablet_copy_client.cc:523] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:02.941532 26767 tablet_bootstrap.cc:492] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap starting.
I20240501 08:18:02.946190 26767 tablet_bootstrap.cc:492] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:02.946491 26767 tablet_bootstrap.cc:492] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: Bootstrap complete.
I20240501 08:18:02.946678 26767 ts_tablet_manager.cc:1367] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:18:02.947276 26767 raft_consensus.cc:348] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.947510 26767 raft_consensus.cc:729] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 946da6d9a36a48b9b67c494ccc6df85e, State: Initialized, Role: LEARNER
I20240501 08:18:02.947691 26767 consensus_queue.cc:260] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: true } }
I20240501 08:18:02.948247 26767 ts_tablet_manager.cc:1398] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20240501 08:18:02.948848 25697 tablet_copy_service.cc:342] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Request end of tablet copy session 946da6d9a36a48b9b67c494ccc6df85e-f14ed0c6ed914ce79e58ecb219b159dd received from {username='slave'} at 127.2.248.131:58505
I20240501 08:18:02.948956 25697 tablet_copy_service.cc:433] P 4bbaf16fd6f6495ea9ba00f007ad03cd: ending tablet copy session 946da6d9a36a48b9b67c494ccc6df85e-f14ed0c6ed914ce79e58ecb219b159dd on tablet f14ed0c6ed914ce79e58ecb219b159dd with peer 946da6d9a36a48b9b67c494ccc6df85e
I20240501 08:18:02.959528 26761 ts_tablet_manager.cc:902] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: Initiating tablet copy from peer 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:02.961807 26761 tablet_copy_client.cc:308] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:40347
I20240501 08:18:02.967204 25697 tablet_copy_service.cc:140] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Received BeginTabletCopySession request for tablet 5394e26d870c48969f2d8670c188691a from peer 6a94029ae37f45e39162bb542ad7a97e ({username='slave'} at 127.2.248.132:51513)
I20240501 08:18:02.967343 25697 tablet_copy_service.cc:161] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Beginning new tablet copy session on tablet 5394e26d870c48969f2d8670c188691a from peer 6a94029ae37f45e39162bb542ad7a97e at {username='slave'} at 127.2.248.132:51513: session id = 6a94029ae37f45e39162bb542ad7a97e-5394e26d870c48969f2d8670c188691a
I20240501 08:18:02.970048 25697 tablet_copy_source_session.cc:205] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:02.971118 26761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5394e26d870c48969f2d8670c188691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:02.976109 26761 tablet_copy_client.cc:783] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:02.976328 26761 tablet_copy_client.cc:647] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:02.977859 26761 tablet_copy_client.cc:523] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:02.981066 26761 tablet_bootstrap.cc:492] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap starting.
I20240501 08:18:02.986151 26761 tablet_bootstrap.cc:492] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:02.986506 26761 tablet_bootstrap.cc:492] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap complete.
I20240501 08:18:02.986732 26761 ts_tablet_manager.cc:1367] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240501 08:18:02.987738 26761 raft_consensus.cc:348] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:02.987998 26817 ts_tablet_manager.cc:902] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: Initiating tablet copy from peer 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:02.988004 26761 raft_consensus.cc:729] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a94029ae37f45e39162bb542ad7a97e, State: Initialized, Role: LEARNER
I20240501 08:18:02.988322 26761 consensus_queue.cc:260] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:02.989432 26761 ts_tablet_manager.cc:1398] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:18:02.992363 25697 tablet_copy_service.cc:342] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Request end of tablet copy session 6a94029ae37f45e39162bb542ad7a97e-5394e26d870c48969f2d8670c188691a received from {username='slave'} at 127.2.248.132:51513
I20240501 08:18:02.992547 25697 tablet_copy_service.cc:433] P 4bbaf16fd6f6495ea9ba00f007ad03cd: ending tablet copy session 6a94029ae37f45e39162bb542ad7a97e-5394e26d870c48969f2d8670c188691a on tablet 5394e26d870c48969f2d8670c188691a with peer 6a94029ae37f45e39162bb542ad7a97e
I20240501 08:18:02.994210 26817 tablet_copy_client.cc:308] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:40347
I20240501 08:18:02.994658 25697 tablet_copy_service.cc:140] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Received BeginTabletCopySession request for tablet a7c9814a54714f0d99d2d2a16e46ea61 from peer 6a94029ae37f45e39162bb542ad7a97e ({username='slave'} at 127.2.248.132:51513)
I20240501 08:18:02.994778 25697 tablet_copy_service.cc:161] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Beginning new tablet copy session on tablet a7c9814a54714f0d99d2d2a16e46ea61 from peer 6a94029ae37f45e39162bb542ad7a97e at {username='slave'} at 127.2.248.132:51513: session id = 6a94029ae37f45e39162bb542ad7a97e-a7c9814a54714f0d99d2d2a16e46ea61
I20240501 08:18:02.997001 25697 tablet_copy_source_session.cc:205] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:02.998106 26817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c9814a54714f0d99d2d2a16e46ea61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:03.003031 26817 tablet_copy_client.cc:783] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:03.003230 26817 tablet_copy_client.cc:647] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:03.004400 26817 tablet_copy_client.cc:523] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:03.006314 26817 tablet_bootstrap.cc:492] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap starting.
I20240501 08:18:03.010775 26817 tablet_bootstrap.cc:492] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:03.011104 26817 tablet_bootstrap.cc:492] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: Bootstrap complete.
I20240501 08:18:03.011258 26817 ts_tablet_manager.cc:1367] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240501 08:18:03.011797 26817 raft_consensus.cc:348] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:03.011930 26817 raft_consensus.cc:729] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a94029ae37f45e39162bb542ad7a97e, State: Initialized, Role: LEARNER
I20240501 08:18:03.012066 26817 consensus_queue.cc:260] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: true } }
I20240501 08:18:03.012516 26817 ts_tablet_manager.cc:1398] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20240501 08:18:03.013247 25697 tablet_copy_service.cc:342] P 4bbaf16fd6f6495ea9ba00f007ad03cd: Request end of tablet copy session 6a94029ae37f45e39162bb542ad7a97e-a7c9814a54714f0d99d2d2a16e46ea61 received from {username='slave'} at 127.2.248.132:51513
I20240501 08:18:03.013348 25697 tablet_copy_service.cc:433] P 4bbaf16fd6f6495ea9ba00f007ad03cd: ending tablet copy session 6a94029ae37f45e39162bb542ad7a97e-a7c9814a54714f0d99d2d2a16e46ea61 on tablet a7c9814a54714f0d99d2d2a16e46ea61 with peer 6a94029ae37f45e39162bb542ad7a97e
I20240501 08:18:03.237829 26500 raft_consensus.cc:1203] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240501 08:18:03.338227 26369 raft_consensus.cc:1203] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240501 08:18:03.394416 26500 raft_consensus.cc:1203] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 08:18:03.407372 26369 raft_consensus.cc:1203] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 08:18:03.449688 26369 raft_consensus.cc:1203] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 08:18:03.559003 26500 raft_consensus.cc:1203] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240501 08:18:03.677572 26848 raft_consensus.cc:1050] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: attempting to promote NON_VOTER 6a94029ae37f45e39162bb542ad7a97e to VOTER
I20240501 08:18:03.678350 26848 consensus_queue.cc:237] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:03.680178 26500 raft_consensus.cc:1261] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Refusing update from remote peer a839e332a8bd470ab44d040cd50c81e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:03.680750 26846 consensus_queue.cc:1035] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:03.683727 26848 raft_consensus.cc:2932] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:03.684460 26500 raft_consensus.cc:2932] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:03.687990 25533 catalog_manager.cc:5484] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: config changed from index 3 to 4, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:03.776566 25808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12b5316170e144648629fa542cc3d49c"
dest_uuid: "a839e332a8bd470ab44d040cd50c81e6"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41338
I20240501 08:18:03.776778 25808 raft_consensus.cc:595] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:03.906317 26860 raft_consensus.cc:1050] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: attempting to promote NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e to VOTER
I20240501 08:18:03.907052 26860 consensus_queue.cc:237] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:03.908874 26369 raft_consensus.cc:1261] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Refusing update from remote peer 4bbaf16fd6f6495ea9ba00f007ad03cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:18:03.909485 26861 consensus_queue.cc:1035] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:18:03.912698 26860 raft_consensus.cc:2932] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:03.913415 26369 raft_consensus.cc:2932] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:03.917493 25534 catalog_manager.cc:5484] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index 2 to 3, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:03.942986 26661 raft_consensus.cc:1050] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: attempting to promote NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e to VOTER
I20240501 08:18:03.943454 26661 consensus_queue.cc:237] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:03.945016 26369 raft_consensus.cc:1261] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Refusing update from remote peer a839e332a8bd470ab44d040cd50c81e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:03.945456 26848 consensus_queue.cc:1035] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:03.947662 26661 raft_consensus.cc:2932] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:03.948194 26369 raft_consensus.cc:2932] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:03.951592 25533 catalog_manager.cc:5484] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 reported cstate change: config changed from index 3 to 4, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a839e332a8bd470ab44d040cd50c81e6" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:03.971468 26673 raft_consensus.cc:1050] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: attempting to promote NON_VOTER 6a94029ae37f45e39162bb542ad7a97e to VOTER
I20240501 08:18:03.971973 26673 consensus_queue.cc:237] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:03.973678 26500 raft_consensus.cc:1261] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Refusing update from remote peer 4bbaf16fd6f6495ea9ba00f007ad03cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:18:03.974103 26860 consensus_queue.cc:1035] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:18:03.976235 26673 raft_consensus.cc:2932] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:03.976740 26500 raft_consensus.cc:2932] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:03.980319 25534 catalog_manager.cc:5484] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index 2 to 3, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.003196 25677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5394e26d870c48969f2d8670c188691a"
dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43158
I20240501 08:18:04.003384 25677 raft_consensus.cc:595] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.020032 26860 raft_consensus.cc:1050] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: attempting to promote NON_VOTER 946da6d9a36a48b9b67c494ccc6df85e to VOTER
I20240501 08:18:04.020579 26860 consensus_queue.cc:237] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.022126 26369 raft_consensus.cc:1261] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 1 LEARNER]: Refusing update from remote peer 4bbaf16fd6f6495ea9ba00f007ad03cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:18:04.022609 26673 consensus_queue.cc:1035] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:18:04.024873 26860 raft_consensus.cc:2932] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:04.025462 26369 raft_consensus.cc:2932] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:04.029034 25533 catalog_manager.cc:5484] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index 2 to 3, 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.032451 25808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d81f2b5e8ee6478888520c203baef7ee"
dest_uuid: "a839e332a8bd470ab44d040cd50c81e6"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41354
I20240501 08:18:04.032626 25808 raft_consensus.cc:595] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.050626 25677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9db572dece424334958f71a54c91c779"
dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43174
I20240501 08:18:04.050802 25677 raft_consensus.cc:595] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.064029 26846 raft_consensus.cc:979] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: : Instructing follower 6a94029ae37f45e39162bb542ad7a97e to start an election
I20240501 08:18:04.064256 26846 raft_consensus.cc:1067] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Signalling peer 6a94029ae37f45e39162bb542ad7a97e to start an election
I20240501 08:18:04.065168 26500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12b5316170e144648629fa542cc3d49c"
dest_uuid: "6a94029ae37f45e39162bb542ad7a97e"
 from {username='slave'} at 127.2.248.130:41263
I20240501 08:18:04.065477 26500 raft_consensus.cc:482] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:04.065667 26500 raft_consensus.cc:3037] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.068249 26500 raft_consensus.cc:504] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.069500 26500 leader_election.cc:288] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [CANDIDATE]: Term 2 election: Requested vote from peers a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130:45313)
I20240501 08:18:04.070120 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b5316170e144648629fa542cc3d49c" candidate_uuid: "6a94029ae37f45e39162bb542ad7a97e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a839e332a8bd470ab44d040cd50c81e6"
I20240501 08:18:04.070158 25807 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12b5316170e144648629fa542cc3d49c"
dest_uuid: "a839e332a8bd470ab44d040cd50c81e6"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41374
I20240501 08:18:04.070384 25807 raft_consensus.cc:595] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.070575 25808 raft_consensus.cc:3032] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:04.070724 25808 raft_consensus.cc:729] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:18:04.071017 25808 consensus_queue.cc:260] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.071312 25808 raft_consensus.cc:3037] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.074384 25808 raft_consensus.cc:2454] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a94029ae37f45e39162bb542ad7a97e in term 2.
I20240501 08:18:04.074926 26436 leader_election.cc:302] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6a94029ae37f45e39162bb542ad7a97e, a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:18:04.075265 26774 raft_consensus.cc:2781] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:04.075657 26774 raft_consensus.cc:686] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 2 LEADER]: Becoming Leader. State: Replica: 6a94029ae37f45e39162bb542ad7a97e, State: Running, Role: LEADER
I20240501 08:18:04.076174 26774 consensus_queue.cc:237] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.079694 25534 catalog_manager.cc:5484] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e reported cstate change: term changed from 1 to 2, leader changed from a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130) to 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132). New cstate: current_term: 2 leader_uuid: "6a94029ae37f45e39162bb542ad7a97e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:18:04.090366 25520 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 12b5316170e144648629fa542cc3d49c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132:35633) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 08:18:04.091531 25677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f14ed0c6ed914ce79e58ecb219b159dd"
dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43186
I20240501 08:18:04.091675 25677 raft_consensus.cc:595] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.179404 26860 raft_consensus.cc:1050] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: attempting to promote NON_VOTER 6a94029ae37f45e39162bb542ad7a97e to VOTER
I20240501 08:18:04.179911 26860 consensus_queue.cc:237] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.181600 26500 raft_consensus.cc:1261] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 1 LEARNER]: Refusing update from remote peer 4bbaf16fd6f6495ea9ba00f007ad03cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:18:04.182088 26860 consensus_queue.cc:1035] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:18:04.184504 26861 raft_consensus.cc:2932] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:04.185130 26500 raft_consensus.cc:2932] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:04.189126 25534 catalog_manager.cc:5484] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd reported cstate change: config changed from index 2 to 3, 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.310390 25677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5394e26d870c48969f2d8670c188691a"
dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43206
I20240501 08:18:04.310599 25677 raft_consensus.cc:595] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.330152 25677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7c9814a54714f0d99d2d2a16e46ea61"
dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43218
I20240501 08:18:04.330355 25677 raft_consensus.cc:595] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.346365 25808 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d81f2b5e8ee6478888520c203baef7ee"
dest_uuid: "a839e332a8bd470ab44d040cd50c81e6"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41394
I20240501 08:18:04.346542 25808 raft_consensus.cc:595] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.362429 25677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9db572dece424334958f71a54c91c779"
dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43250
I20240501 08:18:04.362601 25677 raft_consensus.cc:595] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.388592 25677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f14ed0c6ed914ce79e58ecb219b159dd"
dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43272
I20240501 08:18:04.388806 25677 raft_consensus.cc:595] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:04.410879 26846 raft_consensus.cc:979] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: : Instructing follower 946da6d9a36a48b9b67c494ccc6df85e to start an election
I20240501 08:18:04.411046 26846 raft_consensus.cc:1067] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Signalling peer 946da6d9a36a48b9b67c494ccc6df85e to start an election
I20240501 08:18:04.411653 26369 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d81f2b5e8ee6478888520c203baef7ee"
dest_uuid: "946da6d9a36a48b9b67c494ccc6df85e"
 from {username='slave'} at 127.2.248.130:39651
I20240501 08:18:04.411916 26369 raft_consensus.cc:482] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:04.412070 26369 raft_consensus.cc:3037] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.414055 26369 raft_consensus.cc:504] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.415215 26369 leader_election.cc:288] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [CANDIDATE]: Term 2 election: Requested vote from peers a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130:45313)
I20240501 08:18:04.415699 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f2b5e8ee6478888520c203baef7ee" candidate_uuid: "946da6d9a36a48b9b67c494ccc6df85e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a839e332a8bd470ab44d040cd50c81e6"
I20240501 08:18:04.415910 25808 raft_consensus.cc:3032] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:04.415998 25808 raft_consensus.cc:729] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a839e332a8bd470ab44d040cd50c81e6, State: Running, Role: LEADER
I20240501 08:18:04.416262 25808 consensus_queue.cc:260] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.416527 25808 raft_consensus.cc:3037] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.418303 25808 raft_consensus.cc:2454] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946da6d9a36a48b9b67c494ccc6df85e in term 2.
I20240501 08:18:04.418792 26305 leader_election.cc:302] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 946da6d9a36a48b9b67c494ccc6df85e, a839e332a8bd470ab44d040cd50c81e6; no voters: 
I20240501 08:18:04.419121 26776 raft_consensus.cc:2781] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:04.419497 26776 raft_consensus.cc:686] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 2 LEADER]: Becoming Leader. State: Replica: 946da6d9a36a48b9b67c494ccc6df85e, State: Running, Role: LEADER
I20240501 08:18:04.419906 26776 consensus_queue.cc:237] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.423032 25533 catalog_manager.cc:5484] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e reported cstate change: term changed from 1 to 2, leader changed from a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130) to 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131). New cstate: current_term: 2 leader_uuid: "946da6d9a36a48b9b67c494ccc6df85e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.449580 26673 raft_consensus.cc:979] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: : Instructing follower 946da6d9a36a48b9b67c494ccc6df85e to start an election
I20240501 08:18:04.449748 26673 raft_consensus.cc:1067] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Signalling peer 946da6d9a36a48b9b67c494ccc6df85e to start an election
I20240501 08:18:04.450512 26369 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f14ed0c6ed914ce79e58ecb219b159dd"
dest_uuid: "946da6d9a36a48b9b67c494ccc6df85e"
 from {username='slave'} at 127.2.248.129:54177
I20240501 08:18:04.450673 26369 raft_consensus.cc:482] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:04.450775 26369 raft_consensus.cc:3037] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.452672 26369 raft_consensus.cc:504] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.453171 26369 leader_election.cc:288] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [CANDIDATE]: Term 2 election: Requested vote from peers 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:04.453765 25677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14ed0c6ed914ce79e58ecb219b159dd" candidate_uuid: "946da6d9a36a48b9b67c494ccc6df85e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
I20240501 08:18:04.454016 25677 raft_consensus.cc:3032] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:04.454113 25677 raft_consensus.cc:729] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:18:04.454356 25677 consensus_queue.cc:260] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.454672 25677 raft_consensus.cc:3037] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.456552 25677 raft_consensus.cc:2454] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946da6d9a36a48b9b67c494ccc6df85e in term 2.
I20240501 08:18:04.456946 26303 leader_election.cc:302] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd, 946da6d9a36a48b9b67c494ccc6df85e; no voters: 
I20240501 08:18:04.457208 26776 raft_consensus.cc:2781] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:04.457386 26776 raft_consensus.cc:686] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 2 LEADER]: Becoming Leader. State: Replica: 946da6d9a36a48b9b67c494ccc6df85e, State: Running, Role: LEADER
I20240501 08:18:04.457664 26776 consensus_queue.cc:237] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.459582 25533 catalog_manager.cc:5484] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e reported cstate change: term changed from 1 to 2, leader changed from 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) to 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131). New cstate: current_term: 2 leader_uuid: "946da6d9a36a48b9b67c494ccc6df85e" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.487390 26861 raft_consensus.cc:979] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: : Instructing follower 946da6d9a36a48b9b67c494ccc6df85e to start an election
I20240501 08:18:04.487527 26861 raft_consensus.cc:1067] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Signalling peer 946da6d9a36a48b9b67c494ccc6df85e to start an election
I20240501 08:18:04.488080 26369 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9db572dece424334958f71a54c91c779"
dest_uuid: "946da6d9a36a48b9b67c494ccc6df85e"
 from {username='slave'} at 127.2.248.129:54177
I20240501 08:18:04.488260 26369 raft_consensus.cc:482] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:04.488356 26369 raft_consensus.cc:3037] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.490170 26369 raft_consensus.cc:504] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.490551 26369 leader_election.cc:288] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [CANDIDATE]: Term 2 election: Requested vote from peers 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:04.491039 25677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db572dece424334958f71a54c91c779" candidate_uuid: "946da6d9a36a48b9b67c494ccc6df85e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
I20240501 08:18:04.491210 25677 raft_consensus.cc:3032] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:04.491302 25677 raft_consensus.cc:729] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:18:04.491531 25677 consensus_queue.cc:260] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.491811 25677 raft_consensus.cc:3037] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.493510 25677 raft_consensus.cc:2454] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946da6d9a36a48b9b67c494ccc6df85e in term 2.
I20240501 08:18:04.493849 26303 leader_election.cc:302] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd, 946da6d9a36a48b9b67c494ccc6df85e; no voters: 
I20240501 08:18:04.494184 26776 raft_consensus.cc:2781] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:04.494309 26776 raft_consensus.cc:686] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 2 LEADER]: Becoming Leader. State: Replica: 946da6d9a36a48b9b67c494ccc6df85e, State: Running, Role: LEADER
I20240501 08:18:04.494552 26776 consensus_queue.cc:237] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:04.496435 25533 catalog_manager.cc:5484] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e reported cstate change: term changed from 1 to 2, leader changed from 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) to 946da6d9a36a48b9b67c494ccc6df85e (127.2.248.131). New cstate: current_term: 2 leader_uuid: "946da6d9a36a48b9b67c494ccc6df85e" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.506675 25808 raft_consensus.cc:1261] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Refusing update from remote peer 6a94029ae37f45e39162bb542ad7a97e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:18:04.507329 26774 consensus_queue.cc:1035] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:04.522403 26673 raft_consensus.cc:979] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: : Instructing follower 6a94029ae37f45e39162bb542ad7a97e to start an election
I20240501 08:18:04.522567 26673 raft_consensus.cc:1067] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Signalling peer 6a94029ae37f45e39162bb542ad7a97e to start an election
I20240501 08:18:04.523170 26500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5394e26d870c48969f2d8670c188691a"
dest_uuid: "6a94029ae37f45e39162bb542ad7a97e"
 from {username='slave'} at 127.2.248.129:40621
I20240501 08:18:04.523319 26500 raft_consensus.cc:482] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:04.523406 26500 raft_consensus.cc:3037] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.525166 26500 raft_consensus.cc:504] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.525632 26500 leader_election.cc:288] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [CANDIDATE]: Term 2 election: Requested vote from peers 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:04.526201 25677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5394e26d870c48969f2d8670c188691a" candidate_uuid: "6a94029ae37f45e39162bb542ad7a97e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
I20240501 08:18:04.526351 25677 raft_consensus.cc:3032] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:04.526445 25677 raft_consensus.cc:729] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:18:04.526664 25677 consensus_queue.cc:260] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.526912 25677 raft_consensus.cc:3037] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.528666 25677 raft_consensus.cc:2454] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a94029ae37f45e39162bb542ad7a97e in term 2.
I20240501 08:18:04.529045 26434 leader_election.cc:302] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd, 6a94029ae37f45e39162bb542ad7a97e; no voters: 
I20240501 08:18:04.529322 27060 raft_consensus.cc:2781] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:04.529453 27060 raft_consensus.cc:686] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 2 LEADER]: Becoming Leader. State: Replica: 6a94029ae37f45e39162bb542ad7a97e, State: Running, Role: LEADER
I20240501 08:18:04.529752 27060 consensus_queue.cc:237] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.531688 25534 catalog_manager.cc:5484] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e reported cstate change: term changed from 1 to 2, leader changed from 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) to 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132). New cstate: current_term: 2 leader_uuid: "6a94029ae37f45e39162bb542ad7a97e" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.576129 26861 raft_consensus.cc:979] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: : Instructing follower 6a94029ae37f45e39162bb542ad7a97e to start an election
I20240501 08:18:04.576297 26861 raft_consensus.cc:1067] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Signalling peer 6a94029ae37f45e39162bb542ad7a97e to start an election
I20240501 08:18:04.576907 26500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7c9814a54714f0d99d2d2a16e46ea61"
dest_uuid: "6a94029ae37f45e39162bb542ad7a97e"
 from {username='slave'} at 127.2.248.129:40621
I20240501 08:18:04.577119 26500 raft_consensus.cc:482] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:04.577214 26500 raft_consensus.cc:3037] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.579406 26500 raft_consensus.cc:504] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.579835 26500 leader_election.cc:288] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [CANDIDATE]: Term 2 election: Requested vote from peers 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347)
I20240501 08:18:04.580335 25677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c9814a54714f0d99d2d2a16e46ea61" candidate_uuid: "6a94029ae37f45e39162bb542ad7a97e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd"
I20240501 08:18:04.580492 25677 raft_consensus.cc:3032] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:04.580562 25677 raft_consensus.cc:729] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bbaf16fd6f6495ea9ba00f007ad03cd, State: Running, Role: LEADER
I20240501 08:18:04.580742 25677 consensus_queue.cc:260] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.580993 25677 raft_consensus.cc:3037] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:04.582590 25677 raft_consensus.cc:2454] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a94029ae37f45e39162bb542ad7a97e in term 2.
I20240501 08:18:04.582881 26434 leader_election.cc:302] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4bbaf16fd6f6495ea9ba00f007ad03cd, 6a94029ae37f45e39162bb542ad7a97e; no voters: 
I20240501 08:18:04.583144 27060 raft_consensus.cc:2781] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:04.583283 27060 raft_consensus.cc:686] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 2 LEADER]: Becoming Leader. State: Replica: 6a94029ae37f45e39162bb542ad7a97e, State: Running, Role: LEADER
I20240501 08:18:04.583489 27060 consensus_queue.cc:237] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.585199 25534 catalog_manager.cc:5484] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e reported cstate change: term changed from 1 to 2, leader changed from 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) to 6a94029ae37f45e39162bb542ad7a97e (127.2.248.132). New cstate: current_term: 2 leader_uuid: "6a94029ae37f45e39162bb542ad7a97e" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.752306 26500 consensus_queue.cc:237] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:04.753146 27060 raft_consensus.cc:2932] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:04.757334 25520 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12b5316170e144648629fa542cc3d49c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:18:04.758471 25534 catalog_manager.cc:5484] T 12b5316170e144648629fa542cc3d49c P 6a94029ae37f45e39162bb542ad7a97e reported cstate change: config changed from index 4 to 6, VOTER a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130) evicted. New cstate: current_term: 2 leader_uuid: "6a94029ae37f45e39162bb542ad7a97e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:04.762217 25787 tablet_service.cc:1514] Processing DeleteTablet for tablet 12b5316170e144648629fa542cc3d49c with delete_type TABLET_DATA_TOMBSTONED (TS a839e332a8bd470ab44d040cd50c81e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52330
I20240501 08:18:04.763167 27091 tablet_replica.cc:330] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: stopping tablet replica
I20240501 08:18:04.763581 27091 raft_consensus.cc:2229] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:04.763839 27091 raft_consensus.cc:2258] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:04.765075 27091 ts_tablet_manager.cc:1875] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:04.769239 27091 ts_tablet_manager.cc:1888] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:18:04.769366 27091 log.cc:1196] T 12b5316170e144648629fa542cc3d49c P a839e332a8bd470ab44d040cd50c81e6: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/12b5316170e144648629fa542cc3d49c
I20240501 08:18:04.770143 25523 catalog_manager.cc:4830] TS a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130:45313): tablet 12b5316170e144648629fa542cc3d49c (table rebalance_test_table_2 [id=1459d36a672c447e9b6cb0bfbb3869e0]) successfully deleted
I20240501 08:18:04.980903 25808 raft_consensus.cc:1261] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Refusing update from remote peer 946da6d9a36a48b9b67c494ccc6df85e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:18:04.981559 26776 consensus_queue.cc:1035] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a839e332a8bd470ab44d040cd50c81e6" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 45313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:05.035569 25677 raft_consensus.cc:1261] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Refusing update from remote peer 946da6d9a36a48b9b67c494ccc6df85e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:18:05.036372 27056 consensus_queue.cc:1035] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:05.073689 25677 raft_consensus.cc:1261] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Refusing update from remote peer 946da6d9a36a48b9b67c494ccc6df85e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:18:05.074309 27056 consensus_queue.cc:1035] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:05.088621 26368 consensus_queue.cc:237] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:05.089433 27056 raft_consensus.cc:2932] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:05.091933 25523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d81f2b5e8ee6478888520c203baef7ee with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:18:05.093137 25534 catalog_manager.cc:5484] T d81f2b5e8ee6478888520c203baef7ee P 946da6d9a36a48b9b67c494ccc6df85e reported cstate change: config changed from index 4 to 6, VOTER a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130) evicted. New cstate: current_term: 2 leader_uuid: "946da6d9a36a48b9b67c494ccc6df85e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:05.096014 25787 tablet_service.cc:1514] Processing DeleteTablet for tablet d81f2b5e8ee6478888520c203baef7ee with delete_type TABLET_DATA_TOMBSTONED (TS a839e332a8bd470ab44d040cd50c81e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52330
I20240501 08:18:05.096321 27091 tablet_replica.cc:330] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: stopping tablet replica
I20240501 08:18:05.096505 27091 raft_consensus.cc:2229] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:05.096694 27091 raft_consensus.cc:2258] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:05.097374 27091 ts_tablet_manager.cc:1875] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:05.101337 27091 ts_tablet_manager.cc:1888] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:18:05.101429 27091 log.cc:1196] T d81f2b5e8ee6478888520c203baef7ee P a839e332a8bd470ab44d040cd50c81e6: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/d81f2b5e8ee6478888520c203baef7ee
I20240501 08:18:05.101985 25523 catalog_manager.cc:4830] TS a839e332a8bd470ab44d040cd50c81e6 (127.2.248.130:45313): tablet d81f2b5e8ee6478888520c203baef7ee (table rebalance_test_table_0 [id=8836558475de402991d6ab54c997ba6b]) successfully deleted
I20240501 08:18:05.113875 25677 raft_consensus.cc:1261] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Refusing update from remote peer 6a94029ae37f45e39162bb542ad7a97e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:18:05.114424 27060 consensus_queue.cc:1035] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:05.137746 26368 consensus_queue.cc:237] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:05.138593 26776 raft_consensus.cc:2932] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:05.141141 25523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9db572dece424334958f71a54c91c779 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:18:05.142546 25533 catalog_manager.cc:5484] T 9db572dece424334958f71a54c91c779 P 946da6d9a36a48b9b67c494ccc6df85e reported cstate change: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "946da6d9a36a48b9b67c494ccc6df85e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:05.145959 25656 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db572dece424334958f71a54c91c779 with delete_type TABLET_DATA_TOMBSTONED (TS 4bbaf16fd6f6495ea9ba00f007ad03cd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55342
I20240501 08:18:05.146775 27154 tablet_replica.cc:330] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: stopping tablet replica
I20240501 08:18:05.147150 27154 raft_consensus.cc:2229] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:05.147387 27154 raft_consensus.cc:2258] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:05.148681 27154 ts_tablet_manager.cc:1875] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:05.152997 27154 ts_tablet_manager.cc:1888] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:18:05.153120 27154 log.cc:1196] T 9db572dece424334958f71a54c91c779 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/9db572dece424334958f71a54c91c779
I20240501 08:18:05.153870 25521 catalog_manager.cc:4830] TS 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347): tablet 9db572dece424334958f71a54c91c779 (table rebalance_test_table_2 [id=1459d36a672c447e9b6cb0bfbb3869e0]) successfully deleted
I20240501 08:18:05.181572 26500 consensus_queue.cc:237] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:05.182345 27060 raft_consensus.cc:2932] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:05.185312 25520 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5394e26d870c48969f2d8670c188691a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:18:05.186596 25533 catalog_manager.cc:5484] T 5394e26d870c48969f2d8670c188691a P 6a94029ae37f45e39162bb542ad7a97e reported cstate change: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "6a94029ae37f45e39162bb542ad7a97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:05.190659 25656 tablet_service.cc:1514] Processing DeleteTablet for tablet 5394e26d870c48969f2d8670c188691a with delete_type TABLET_DATA_TOMBSTONED (TS 4bbaf16fd6f6495ea9ba00f007ad03cd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55342
I20240501 08:18:05.191016 27154 tablet_replica.cc:330] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: stopping tablet replica
I20240501 08:18:05.191222 27154 raft_consensus.cc:2229] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:05.191430 27154 raft_consensus.cc:2258] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:05.192176 27154 ts_tablet_manager.cc:1875] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:05.196206 27154 ts_tablet_manager.cc:1888] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:18:05.196316 27154 log.cc:1196] T 5394e26d870c48969f2d8670c188691a P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/5394e26d870c48969f2d8670c188691a
I20240501 08:18:05.197088 25521 catalog_manager.cc:4830] TS 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347): tablet 5394e26d870c48969f2d8670c188691a (table TestTable [id=112434fea13441cf93ed191eddc48203]) successfully deleted
I20240501 08:18:05.203125 25677 raft_consensus.cc:1261] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Refusing update from remote peer 6a94029ae37f45e39162bb542ad7a97e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:18:05.203621 26774 consensus_queue.cc:1035] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbaf16fd6f6495ea9ba00f007ad03cd" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 40347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:05.220135 26500 consensus_queue.cc:237] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } }
I20240501 08:18:05.221043 27181 raft_consensus.cc:2932] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } } }
I20240501 08:18:05.225317 25520 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7c9814a54714f0d99d2d2a16e46ea61 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:18:05.226152 25533 catalog_manager.cc:5484] T a7c9814a54714f0d99d2d2a16e46ea61 P 6a94029ae37f45e39162bb542ad7a97e reported cstate change: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "6a94029ae37f45e39162bb542ad7a97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a94029ae37f45e39162bb542ad7a97e" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:05.229876 25656 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c9814a54714f0d99d2d2a16e46ea61 with delete_type TABLET_DATA_TOMBSTONED (TS 4bbaf16fd6f6495ea9ba00f007ad03cd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55342
I20240501 08:18:05.230192 27154 tablet_replica.cc:330] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: stopping tablet replica
I20240501 08:18:05.230378 27154 raft_consensus.cc:2229] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:05.230584 27154 raft_consensus.cc:2258] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:05.231432 27154 ts_tablet_manager.cc:1875] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:05.237021 27154 ts_tablet_manager.cc:1888] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:18:05.237114 27154 log.cc:1196] T a7c9814a54714f0d99d2d2a16e46ea61 P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/a7c9814a54714f0d99d2d2a16e46ea61
I20240501 08:18:05.237757 25521 catalog_manager.cc:4830] TS 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347): tablet a7c9814a54714f0d99d2d2a16e46ea61 (table rebalance_test_table_3 [id=c2b5627bcd7546e793340948e2c12233]) successfully deleted
I20240501 08:18:05.582048 26368 consensus_queue.cc:237] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } }
I20240501 08:18:05.582937 27056 raft_consensus.cc:2932] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } } }
I20240501 08:18:05.585788 25523 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f14ed0c6ed914ce79e58ecb219b159dd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240501 08:18:05.587195 25533 catalog_manager.cc:5484] T f14ed0c6ed914ce79e58ecb219b159dd P 946da6d9a36a48b9b67c494ccc6df85e reported cstate change: config changed from index 3 to 5, VOTER 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "946da6d9a36a48b9b67c494ccc6df85e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "946da6d9a36a48b9b67c494ccc6df85e" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 45885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:05.590318 25656 tablet_service.cc:1514] Processing DeleteTablet for tablet f14ed0c6ed914ce79e58ecb219b159dd with delete_type TABLET_DATA_TOMBSTONED (TS 4bbaf16fd6f6495ea9ba00f007ad03cd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55342
I20240501 08:18:05.590631 27154 tablet_replica.cc:330] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: stopping tablet replica
I20240501 08:18:05.590786 27154 raft_consensus.cc:2229] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:05.590932 27154 raft_consensus.cc:2258] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:05.591538 27154 ts_tablet_manager.cc:1875] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:05.595656 27154 ts_tablet_manager.cc:1888] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:18:05.595752 27154 log.cc:1196] T f14ed0c6ed914ce79e58ecb219b159dd P 4bbaf16fd6f6495ea9ba00f007ad03cd: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/f14ed0c6ed914ce79e58ecb219b159dd
I20240501 08:18:05.596421 25521 catalog_manager.cc:4830] TS 4bbaf16fd6f6495ea9ba00f007ad03cd (127.2.248.129:40347): tablet f14ed0c6ed914ce79e58ecb219b159dd (table rebalance_test_table_1 [id=d7d7552aef85466e9309288251370a5c]) successfully deleted
I20240501 08:18:05.764344 25656 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:05.769369 26480 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:05.780354 25787 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:05.792279 26349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:05.919121 26349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:05.919464 25656 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:05.920835 25787 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:05.930754 26480 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:06.194839 25656 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:06.197391 26480 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:06.199030 26349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:18:06.207001 25787 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 04ebf9d3a1754802ad27defb32c04321 | 127.2.248.190:45945 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                               |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                              | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                 | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                              | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                              | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                              | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                               | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:44779 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 4bbaf16fd6f6495ea9ba00f007ad03cd | 127.2.248.129:40347 | HEALTHY | <none>   |       3        |       0
 6a94029ae37f45e39162bb542ad7a97e | 127.2.248.132:35633 | HEALTHY | <none>   |       3        |       0
 946da6d9a36a48b9b67c494ccc6df85e | 127.2.248.131:45885 | HEALTHY | <none>   |       3        |       0
 a839e332a8bd470ab44d040cd50c81e6 | 127.2.248.130:45313 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                             |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                           | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.2.248.129                                                                                                                                                                                 | experimental          | 127.2.248.129:40347
 local_ip_for_outbound_sockets            | 127.2.248.130                                                                                                                                                                                 | experimental          | 127.2.248.130:45313
 local_ip_for_outbound_sockets            | 127.2.248.131                                                                                                                                                                                 | experimental          | 127.2.248.131:45885
 local_ip_for_outbound_sockets            | 127.2.248.132                                                                                                                                                                                 | experimental          | 127.2.248.132:35633
 never_fsync                              | true                                                                                                                                                                                          | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                             | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                          | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                          | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                            | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.2.248.129:40347
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.2.248.130:45313
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.2.248.131:45885
 server_dump_info_path                    | /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.2.248.132:35633

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.2.248.148:44779 | all 4 server(s) checked
 time_source         | builtin             | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 3
 First Quartile | 3
 Median         | 3
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240501 08:18:06.261240  3042 log_verifier.cc:126] Checking tablet 108a1ac12fa04ce9a525a20d1680f4bd
I20240501 08:18:06.269332  3042 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 108a1ac12fa04ce9a525a20d1680f4bd
I20240501 08:18:06.269435  3042 log_verifier.cc:126] Checking tablet 12b5316170e144648629fa542cc3d49c
I20240501 08:18:06.278640  3042 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 12b5316170e144648629fa542cc3d49c
I20240501 08:18:06.278748  3042 log_verifier.cc:126] Checking tablet 2c511b81062a47b388bc886786bffca0
I20240501 08:18:06.286319  3042 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2c511b81062a47b388bc886786bffca0
I20240501 08:18:06.286394  3042 log_verifier.cc:126] Checking tablet 5394e26d870c48969f2d8670c188691a
I20240501 08:18:06.294628  3042 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 5394e26d870c48969f2d8670c188691a
I20240501 08:18:06.294715  3042 log_verifier.cc:126] Checking tablet 64562a17576d48e7b8e2b079d24ec1b4
I20240501 08:18:06.302333  3042 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 64562a17576d48e7b8e2b079d24ec1b4
I20240501 08:18:06.302415  3042 log_verifier.cc:126] Checking tablet 706f5a79f2f34a21831cdcd9ac6b0aab
I20240501 08:18:06.309903  3042 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 706f5a79f2f34a21831cdcd9ac6b0aab
I20240501 08:18:06.309978  3042 log_verifier.cc:126] Checking tablet 78a27dc2d0d1404a916493386059b301
I20240501 08:18:06.317127  3042 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 78a27dc2d0d1404a916493386059b301
I20240501 08:18:06.317195  3042 log_verifier.cc:126] Checking tablet 9db572dece424334958f71a54c91c779
I20240501 08:18:06.325717  3042 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 9db572dece424334958f71a54c91c779
I20240501 08:18:06.325819  3042 log_verifier.cc:126] Checking tablet a09ad135fef749a1a292070a9a2fdbaa
I20240501 08:18:06.333577  3042 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a09ad135fef749a1a292070a9a2fdbaa
I20240501 08:18:06.333667  3042 log_verifier.cc:126] Checking tablet a7c9814a54714f0d99d2d2a16e46ea61
I20240501 08:18:06.341378  3042 log_verifier.cc:177] Verified matching terms for 5 ops in tablet a7c9814a54714f0d99d2d2a16e46ea61
I20240501 08:18:06.341476  3042 log_verifier.cc:126] Checking tablet b619c0c69bf644c4aff333388d10ea4e
I20240501 08:18:06.349172  3042 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b619c0c69bf644c4aff333388d10ea4e
I20240501 08:18:06.349264  3042 log_verifier.cc:126] Checking tablet d81f2b5e8ee6478888520c203baef7ee
I20240501 08:18:06.357313  3042 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d81f2b5e8ee6478888520c203baef7ee
I20240501 08:18:06.357412  3042 log_verifier.cc:126] Checking tablet f14ed0c6ed914ce79e58ecb219b159dd
I20240501 08:18:06.365259  3042 log_verifier.cc:177] Verified matching terms for 5 ops in tablet f14ed0c6ed914ce79e58ecb219b159dd
I20240501 08:18:06.366125  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 25595
I20240501 08:18:06.367465 25717 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:06.492096  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 25595
I20240501 08:18:06.513794  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 25726
I20240501 08:18:06.514758 25847 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:06.623139  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 25726
I20240501 08:18:06.643779  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 26287
I20240501 08:18:06.650209 26408 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:06.750200  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 26287
I20240501 08:18:06.768909  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 26418
I20240501 08:18:06.774333 26540 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:06.873538  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 26418
I20240501 08:18:06.892984  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 25505
I20240501 08:18:06.894032 25566 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:07.015683  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 25505
2024-05-01T08:18:07Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (15901 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (15901 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240501 08:18:07.072993  3042 test_util.cc:268] Using random seed: 542509762
I20240501 08:18:07.074801  3042 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:18:07.074863  3042 ts_itest-base.cc:113] --------------
I20240501 08:18:07.074908  3042 ts_itest-base.cc:114] 6 tablet servers
I20240501 08:18:07.074959  3042 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:18:07.075011  3042 ts_itest-base.cc:116] --------------
2024-05-01T08:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:18:07Z Disabled control of system clock
I20240501 08:18:07.120898  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:45677
--webserver_interface=127.2.248.190
--webserver_port=0
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.248.190:45677
--location_mapping_cmd=/tmp/dist-test-task3vHYYh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 08:18:07.450598 27315 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 08:18:07.450994 27315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:07.451109 27315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:07.451215 27315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:07.458609 27315 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:07.458720 27315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:18:07.458822 27315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:07.458875 27315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:18:07.458962 27315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:18:07.468065 27315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task3vHYYh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.2.248.190:45677
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.248.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:07.469415 27315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:07.471104 27315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:07.481868 27321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:07.481931 27323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:07.482403 27320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:07.482959 27315 server_base.cc:1034] running on GCE node
I20240501 08:18:07.483624 27315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:07.486452 27315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:07.488006 27315 hybrid_clock.cc:648] HybridClock initialized: now 1714551487487861 us; error 89 us; skew 500 ppm
I20240501 08:18:07.488598 27315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:07.490890 27315 webserver.cc:460] Webserver started at http://127.2.248.190:32921/ using document root <none> and password file <none>
I20240501 08:18:07.491564 27315 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:07.491680 27315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:07.491981 27315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:07.494534 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9039958a12f8452f99a1adde2a3c0c0b"
format_stamp: "Formatted at 2024-05-01 08:18:07 on dist-test-slave-0pdg"
I20240501 08:18:07.495263 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9039958a12f8452f99a1adde2a3c0c0b"
format_stamp: "Formatted at 2024-05-01 08:18:07 on dist-test-slave-0pdg"
I20240501 08:18:07.500108 27315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240501 08:18:07.503239 27329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:07.504459 27315 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:07.504614 27315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9039958a12f8452f99a1adde2a3c0c0b"
format_stamp: "Formatted at 2024-05-01 08:18:07 on dist-test-slave-0pdg"
I20240501 08:18:07.504771 27315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:07.531611 27315 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:07.532446 27315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:07.532738 27315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:07.550045 27315 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.190:45677
I20240501 08:18:07.550129 27380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.190:45677 every 8 connection(s)
I20240501 08:18:07.551795 27315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:18:07.555426 27381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:07.558627  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 27315
I20240501 08:18:07.558848  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:18:07.566056 27381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b: Bootstrap starting.
I20240501 08:18:07.569535 27381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:07.570880 27381 log.cc:824] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:07.573551 27381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b: No bootstrap required, opened a new log
I20240501 08:18:07.579305 27381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9039958a12f8452f99a1adde2a3c0c0b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45677 } }
I20240501 08:18:07.579638 27381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:07.579731 27381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9039958a12f8452f99a1adde2a3c0c0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:07.580281 27381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9039958a12f8452f99a1adde2a3c0c0b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45677 } }
I20240501 08:18:07.580471 27381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:18:07.580585 27381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:18:07.580750 27381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:07.582643 27381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9039958a12f8452f99a1adde2a3c0c0b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45677 } }
I20240501 08:18:07.583096 27381 leader_election.cc:302] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9039958a12f8452f99a1adde2a3c0c0b; no voters: 
I20240501 08:18:07.583636 27381 leader_election.cc:288] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:18:07.583745 27386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:07.584409 27386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [term 1 LEADER]: Becoming Leader. State: Replica: 9039958a12f8452f99a1adde2a3c0c0b, State: Running, Role: LEADER
I20240501 08:18:07.584911 27386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9039958a12f8452f99a1adde2a3c0c0b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45677 } }
I20240501 08:18:07.585546 27381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:18:07.587533 27387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9039958a12f8452f99a1adde2a3c0c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9039958a12f8452f99a1adde2a3c0c0b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45677 } } }
I20240501 08:18:07.587868 27387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [sys.catalog]: This master's current role is: LEADER
I20240501 08:18:07.591951 27388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9039958a12f8452f99a1adde2a3c0c0b. Latest consensus state: current_term: 1 leader_uuid: "9039958a12f8452f99a1adde2a3c0c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9039958a12f8452f99a1adde2a3c0c0b" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 45677 } } }
I20240501 08:18:07.592308 27388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b [sys.catalog]: This master's current role is: LEADER
W20240501 08:18:07.596464 27401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 08:18:07.596565 27401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 08:18:07.596661 27393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:18:07.598718 27393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:18:07.606600 27393 catalog_manager.cc:1347] Generated new cluster ID: 48bc7c3cb7d74e92986d97e383553ccb
I20240501 08:18:07.606710 27393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:18:07.623286 27393 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:18:07.624434 27393 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:18:07.635507 27393 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9039958a12f8452f99a1adde2a3c0c0b: Generated new TSK 0
I20240501 08:18:07.636337 27393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:18:07.646391  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.129:0
--local_ip_for_outbound_sockets=127.2.248.129
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:18:07.974599 27405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:07.974885 27405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:07.975000 27405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:07.982164 27405 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:07.982273 27405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:07.982439 27405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.129
I20240501 08:18:07.990923 27405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.129
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:07.992280 27405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:07.993856 27405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:08.005349 27410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:08.005542 27411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:08.005882 27413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:08.006569 27405 server_base.cc:1034] running on GCE node
I20240501 08:18:08.007110 27405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:08.008049 27405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:08.009310 27405 hybrid_clock.cc:648] HybridClock initialized: now 1714551488009274 us; error 33 us; skew 500 ppm
I20240501 08:18:08.009617 27405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:08.011839 27405 webserver.cc:460] Webserver started at http://127.2.248.129:37703/ using document root <none> and password file <none>
I20240501 08:18:08.012567 27405 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:08.012686 27405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:08.012976 27405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:08.015323 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "64da46caf0204911aba9aba77e5ba045"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.016029 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "64da46caf0204911aba9aba77e5ba045"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.020720 27405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240501 08:18:08.023980 27419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.025305 27405 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:18:08.025457 27405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "64da46caf0204911aba9aba77e5ba045"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.025655 27405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:08.038791 27405 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:08.039630 27405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:08.039917 27405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:08.040930 27405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:08.042452 27405 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:08.042554 27405 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.042676 27405 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:08.042763 27405 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.075769 27405 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.129:44543
I20240501 08:18:08.075822 27531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.129:44543 every 8 connection(s)
I20240501 08:18:08.077021 27405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:18:08.084684  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 27405
I20240501 08:18:08.084897  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:18:08.088713  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.130:0
--local_ip_for_outbound_sockets=127.2.248.130
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:08.089398 27532 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:08.089725 27532 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:08.090587 27532 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:08.137671 27346 ts_manager.cc:194] Registered new tserver with Master: 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543)
I20240501 08:18:08.140014 27346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.129:44849
W20240501 08:18:08.424309 27536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:08.424576 27536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:08.424660 27536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:08.432021 27536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:08.432152 27536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:08.432284 27536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.130
I20240501 08:18:08.440853 27536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.130
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:08.442080 27536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:08.443836 27536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:08.455302 27542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:08.455711 27545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:08.456254 27543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:08.456498 27536 server_base.cc:1034] running on GCE node
I20240501 08:18:08.457036 27536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:08.457919 27536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:08.459115 27536 hybrid_clock.cc:648] HybridClock initialized: now 1714551488459062 us; error 51 us; skew 500 ppm
I20240501 08:18:08.459430 27536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:08.465683 27536 webserver.cc:460] Webserver started at http://127.2.248.130:33627/ using document root <none> and password file <none>
I20240501 08:18:08.466460 27536 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:08.466594 27536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:08.466907 27536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:08.469673 27536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.470453 27536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.475499 27536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240501 08:18:08.478685 27551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.479990 27536 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:08.480159 27536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.480342 27536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:08.495685 27536 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:08.496436 27536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:08.496678 27536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:08.497527 27536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:08.498842 27536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:08.498927 27536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.499042 27536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:08.499112 27536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.529929 27536 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.130:46823
I20240501 08:18:08.529992 27663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.130:46823 every 8 connection(s)
I20240501 08:18:08.531208 27536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:18:08.540069  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 27536
I20240501 08:18:08.540266  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:18:08.542913 27664 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:08.543211 27664 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:08.543483  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:0
--local_ip_for_outbound_sockets=127.2.248.131
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:08.544138 27664 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:08.605536 27346 ts_manager.cc:194] Registered new tserver with Master: dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:08.606467 27346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.130:36357
W20240501 08:18:08.886096 27668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:08.886384 27668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:08.886531 27668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:08.893844 27668 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:08.893986 27668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:08.894140 27668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:18:08.902782 27668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:08.904093 27668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:08.905774 27668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:08.917798 27673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:08.918190 27676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:08.918280 27674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:08.918850 27668 server_base.cc:1034] running on GCE node
I20240501 08:18:08.919414 27668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:08.920423 27668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:08.921622 27668 hybrid_clock.cc:648] HybridClock initialized: now 1714551488921578 us; error 40 us; skew 500 ppm
I20240501 08:18:08.921918 27668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:08.924073 27668 webserver.cc:460] Webserver started at http://127.2.248.131:40587/ using document root <none> and password file <none>
I20240501 08:18:08.924809 27668 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:08.924932 27668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:08.925241 27668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:08.927773 27668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8b39dff53c8344d3af0bf354b31998eb"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.928529 27668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8b39dff53c8344d3af0bf354b31998eb"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.933472 27668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240501 08:18:08.936592 27682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.937810 27668 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.002s
I20240501 08:18:08.937954 27668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8b39dff53c8344d3af0bf354b31998eb"
format_stamp: "Formatted at 2024-05-01 08:18:08 on dist-test-slave-0pdg"
I20240501 08:18:08.938130 27668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:08.955641 27668 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:08.956509 27668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:08.956754 27668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:08.957554 27668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:08.958899 27668 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:08.958990 27668 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.959108 27668 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:08.959173 27668 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:08.989359 27668 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:38383
I20240501 08:18:08.989398 27794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:38383 every 8 connection(s)
I20240501 08:18:08.990723 27668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:18:08.998600  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 27668
I20240501 08:18:08.998755  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:18:09.002424  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:0
--local_ip_for_outbound_sockets=127.2.248.132
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:09.003230 27795 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:09.003491 27795 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:09.004278 27795 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:09.047943 27346 ts_manager.cc:194] Registered new tserver with Master: 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:09.049177 27346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:51871
I20240501 08:18:09.143705 27532 heartbeater.cc:503] Master 127.2.248.190:45677 was elected leader, sending a full tablet report...
W20240501 08:18:09.333997 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:09.334380 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:09.334537 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:09.342612 27799 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:09.342747 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:09.342909 27799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:18:09.351794 27799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:09.353098 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:09.354849 27799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:09.366170 27804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:09.366663 27805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:09.367178 27807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:09.367828 27799 server_base.cc:1034] running on GCE node
I20240501 08:18:09.368446 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:09.369423 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:09.370622 27799 hybrid_clock.cc:648] HybridClock initialized: now 1714551489370577 us; error 43 us; skew 500 ppm
I20240501 08:18:09.370913 27799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:09.373205 27799 webserver.cc:460] Webserver started at http://127.2.248.132:39633/ using document root <none> and password file <none>
I20240501 08:18:09.373924 27799 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:09.374080 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:09.374377 27799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:09.376909 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9a400de32dd143fbbf6485d0ddc1dba7"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:09.377626 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9a400de32dd143fbbf6485d0ddc1dba7"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:09.382576 27799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240501 08:18:09.385923 27813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:09.387205 27799 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:09.387348 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9a400de32dd143fbbf6485d0ddc1dba7"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:09.387523 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:09.404983 27799 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:09.405766 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:09.406008 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:09.406880 27799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:09.408363 27799 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:09.408459 27799 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:09.408582 27799 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:09.408655 27799 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:09.439121 27799 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:34905
I20240501 08:18:09.439165 27925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:34905 every 8 connection(s)
I20240501 08:18:09.440451 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:18:09.443917  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 27799
I20240501 08:18:09.444151  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:18:09.447316  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:0
--local_ip_for_outbound_sockets=127.2.248.133
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:09.453567 27926 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:09.453850 27926 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:09.454623 27926 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:09.499105 27346 ts_manager.cc:194] Registered new tserver with Master: 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132:34905)
I20240501 08:18:09.500370 27346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:47301
I20240501 08:18:09.609171 27664 heartbeater.cc:503] Master 127.2.248.190:45677 was elected leader, sending a full tablet report...
W20240501 08:18:09.775269 27929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:09.775525 27929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:09.775656 27929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:09.782585 27929 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:09.782696 27929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:09.782852 27929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:18:09.791939 27929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:09.793330 27929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:09.794983 27929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:09.806273 27936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:09.806784 27938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:09.806859 27935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:09.807439 27929 server_base.cc:1034] running on GCE node
I20240501 08:18:09.808022 27929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:09.808949 27929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:09.810150 27929 hybrid_clock.cc:648] HybridClock initialized: now 1714551489810092 us; error 44 us; skew 500 ppm
I20240501 08:18:09.810451 27929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:09.812544 27929 webserver.cc:460] Webserver started at http://127.2.248.133:46265/ using document root <none> and password file <none>
I20240501 08:18:09.813247 27929 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:09.813370 27929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:09.813669 27929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:09.816022 27929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b755b190bbfc4bc0a08c28c7263c94cd"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:09.816807 27929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b755b190bbfc4bc0a08c28c7263c94cd"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:09.821597 27929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.003s
I20240501 08:18:09.824689 27944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:09.825886 27929 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:18:09.826036 27929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b755b190bbfc4bc0a08c28c7263c94cd"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:09.826207 27929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:09.844960 27929 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:09.845850 27929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:09.846141 27929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:09.847047 27929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:09.848477 27929 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:09.848587 27929 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:09.848708 27929 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:09.848785 27929 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:09.879199 27929 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:42453
I20240501 08:18:09.879236 28056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:42453 every 8 connection(s)
I20240501 08:18:09.880520 27929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:18:09.886557  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 27929
I20240501 08:18:09.886751  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:18:09.889931  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.134:0
--local_ip_for_outbound_sockets=127.2.248.134
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:09.893397 28057 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:09.893661 28057 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:09.894407 28057 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:09.939013 27346 ts_manager.cc:194] Registered new tserver with Master: b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133:42453)
I20240501 08:18:09.940296 27346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:50607
I20240501 08:18:10.052371 27795 heartbeater.cc:503] Master 127.2.248.190:45677 was elected leader, sending a full tablet report...
W20240501 08:18:10.219058 28060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:10.219312 28060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:10.219396 28060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:10.226560 28060 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:10.226661 28060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:10.226778 28060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.134
I20240501 08:18:10.235433 28060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.248.134
--webserver_port=0
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.134
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:10.236691 28060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:10.238289 28060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:10.249644 28066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:10.250161 28069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:10.250557 28067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:10.250852 28060 server_base.cc:1034] running on GCE node
I20240501 08:18:10.251372 28060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:10.252211 28060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:10.253422 28060 hybrid_clock.cc:648] HybridClock initialized: now 1714551490253388 us; error 34 us; skew 500 ppm
I20240501 08:18:10.253788 28060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:10.255913 28060 webserver.cc:460] Webserver started at http://127.2.248.134:39295/ using document root <none> and password file <none>
I20240501 08:18:10.256633 28060 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:10.256774 28060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:10.257037 28060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:10.259366 28060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "40c65a4181944bf69feec459e131a224"
format_stamp: "Formatted at 2024-05-01 08:18:10 on dist-test-slave-0pdg"
I20240501 08:18:10.260025 28060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "40c65a4181944bf69feec459e131a224"
format_stamp: "Formatted at 2024-05-01 08:18:10 on dist-test-slave-0pdg"
I20240501 08:18:10.264793 28060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.001s
I20240501 08:18:10.267741 28075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:10.269035 28060 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:18:10.269160 28060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "40c65a4181944bf69feec459e131a224"
format_stamp: "Formatted at 2024-05-01 08:18:10 on dist-test-slave-0pdg"
I20240501 08:18:10.269311 28060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:10.305857 28060 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:10.306615 28060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:10.306844 28060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:10.307723 28060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:10.309203 28060 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:10.309280 28060 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:10.309352 28060 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:10.309393 28060 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:10.339844 28060 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.134:41033
I20240501 08:18:10.339917 28187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.134:41033 every 8 connection(s)
I20240501 08:18:10.341243 28060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:18:10.348321  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 28060
I20240501 08:18:10.348500  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:18:10.353585 28188 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:10.353837 28188 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:10.354553 28188 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:10.398213 27346 ts_manager.cc:194] Registered new tserver with Master: 40c65a4181944bf69feec459e131a224 (127.2.248.134:41033)
I20240501 08:18:10.399192 27346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.134:39233
I20240501 08:18:10.401043  3042 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 08:18:10.416960  3042 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 08:18:10.417101  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 27799
I20240501 08:18:10.424069 27921 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:10.500931  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 27799
I20240501 08:18:10.517843  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 27929
I20240501 08:18:10.525322 28052 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:10.605918  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 27929
I20240501 08:18:10.621645  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 28060
I20240501 08:18:10.629762 28183 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:10.710687  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 28060
I20240501 08:18:14.479606 27346 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60802:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 08:18:14.486285 27346 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:18:14.517918 27466 tablet_service.cc:1467] Processing CreateTablet for tablet e41a43a3a85048669f6718a4be9aa382 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.519567 27466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41a43a3a85048669f6718a4be9aa382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.519767 27597 tablet_service.cc:1467] Processing CreateTablet for tablet a8231124b83946e7bbe6184621bd043d (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.519649 27599 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6e4739226a4fc1a8270753768ce569 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.520002 27728 tablet_service.cc:1467] Processing CreateTablet for tablet a8231124b83946e7bbe6184621bd043d (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.520224 27729 tablet_service.cc:1467] Processing CreateTablet for tablet e41a43a3a85048669f6718a4be9aa382 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.521119 27599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6e4739226a4fc1a8270753768ce569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.521109 27598 tablet_service.cc:1467] Processing CreateTablet for tablet e41a43a3a85048669f6718a4be9aa382 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.521430 27728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8231124b83946e7bbe6184621bd043d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.521654 27598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41a43a3a85048669f6718a4be9aa382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.522166 27729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e41a43a3a85048669f6718a4be9aa382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.522532 27730 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6e4739226a4fc1a8270753768ce569 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.523002 27730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6e4739226a4fc1a8270753768ce569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.521047 27465 tablet_service.cc:1467] Processing CreateTablet for tablet a8231124b83946e7bbe6184621bd043d (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.521667 27467 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6e4739226a4fc1a8270753768ce569 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:14.524041 27597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8231124b83946e7bbe6184621bd043d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.524598 27465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8231124b83946e7bbe6184621bd043d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.525758 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6e4739226a4fc1a8270753768ce569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:14.538937 28214 tablet_bootstrap.cc:492] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:14.545325 28214 tablet_bootstrap.cc:654] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.546576 28214 log.cc:824] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:14.547955 28216 tablet_bootstrap.cc:492] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:14.548394 28215 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:14.552060 28215 tablet_bootstrap.cc:654] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.553828 28216 tablet_bootstrap.cc:654] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.554136 28215 log.cc:824] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:14.555136 28216 log.cc:824] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:14.555756 28214 tablet_bootstrap.cc:492] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:14.556042 28214 ts_tablet_manager.cc:1367] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.000s
I20240501 08:18:14.563387 28215 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:14.563793 28215 ts_tablet_manager.cc:1367] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.006s
I20240501 08:18:14.564921 28214 raft_consensus.cc:348] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.565428 28214 raft_consensus.cc:374] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.565553 28214 raft_consensus.cc:729] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.566246 28214 consensus_queue.cc:260] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.573351 28214 ts_tablet_manager.cc:1398] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.017s	user 0.015s	sys 0.001s
I20240501 08:18:14.573737 28214 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:14.574978 28214 tablet_bootstrap.cc:654] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.574913 28215 raft_consensus.cc:348] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.575397 28215 raft_consensus.cc:374] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.575523 28215 raft_consensus.cc:729] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.574064 28216 tablet_bootstrap.cc:492] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:14.576313 28215 consensus_queue.cc:260] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.576489 28216 ts_tablet_manager.cc:1367] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.030s	user 0.014s	sys 0.008s
I20240501 08:18:14.577234 28214 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:14.577452 28214 ts_tablet_manager.cc:1367] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 08:18:14.578505 28214 raft_consensus.cc:348] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.578780 28214 raft_consensus.cc:374] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.578909 28214 raft_consensus.cc:729] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.579074 28215 ts_tablet_manager.cc:1398] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.015s	user 0.015s	sys 0.001s
I20240501 08:18:14.579221 28214 consensus_queue.cc:260] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.579666 28215 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:14.579913 28214 ts_tablet_manager.cc:1398] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:18:14.580214 28214 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:14.581561 28215 tablet_bootstrap.cc:654] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.581743 28214 tablet_bootstrap.cc:654] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
W20240501 08:18:14.582949 27533 tablet.cc:2369] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:18:14.583006 28220 raft_consensus.cc:482] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:14.583235 28220 raft_consensus.cc:504] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.585701 28220 leader_election.cc:288] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:14.588008 28216 raft_consensus.cc:348] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.588573 28216 raft_consensus.cc:374] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.588697 28216 raft_consensus.cc:729] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.589588 28216 consensus_queue.cc:260] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.600039 28216 ts_tablet_manager.cc:1398] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.023s	user 0.015s	sys 0.004s
I20240501 08:18:14.600646 28216 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:14.602896 28216 tablet_bootstrap.cc:654] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.612237 28216 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:14.612409 28216 ts_tablet_manager.cc:1367] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.004s
I20240501 08:18:14.613620 28214 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:14.613855 28214 ts_tablet_manager.cc:1367] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.034s	user 0.002s	sys 0.030s
I20240501 08:18:14.613991 28215 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:14.613963 28216 raft_consensus.cc:348] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.614161 28215 ts_tablet_manager.cc:1367] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.035s	user 0.011s	sys 0.018s
I20240501 08:18:14.614212 28216 raft_consensus.cc:374] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.614310 28216 raft_consensus.cc:729] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.614547 28216 consensus_queue.cc:260] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.615005 28214 raft_consensus.cc:348] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.615208 28216 ts_tablet_manager.cc:1398] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:14.615166 28215 raft_consensus.cc:348] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.615381 28215 raft_consensus.cc:374] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.615340 28214 raft_consensus.cc:374] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.615479 28215 raft_consensus.cc:729] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.615499 28216 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:14.615586 28214 raft_consensus.cc:729] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.615754 28215 consensus_queue.cc:260] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.615970 28214 consensus_queue.cc:260] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.616379 28215 ts_tablet_manager.cc:1398] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:14.616619 28215 tablet_bootstrap.cc:492] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:14.616605 28214 ts_tablet_manager.cc:1398] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:18:14.617311 28216 tablet_bootstrap.cc:654] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.617859 28215 tablet_bootstrap.cc:654] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:14.619853 28215 tablet_bootstrap.cc:492] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:14.619858 28216 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:14.620018 28215 ts_tablet_manager.cc:1367] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:14.620137 28216 ts_tablet_manager.cc:1367] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:18:14.620831 28215 raft_consensus.cc:348] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.620885 28216 raft_consensus.cc:348] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.621104 28215 raft_consensus.cc:374] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.621204 28215 raft_consensus.cc:729] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.621253 28216 raft_consensus.cc:374] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:14.621426 28216 raft_consensus.cc:729] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:14.621466 28215 consensus_queue.cc:260] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.621762 28216 consensus_queue.cc:260] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.622196 28215 ts_tablet_manager.cc:1398] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:18:14.622398 28216 ts_tablet_manager.cc:1398] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:14.626261 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8231124b83946e7bbe6184621bd043d" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045" is_pre_election: true
I20240501 08:18:14.626770 27487 raft_consensus.cc:2454] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:14.627560 27685 leader_election.cc:302] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb; no voters: 
I20240501 08:18:14.628031 28220 raft_consensus.cc:2781] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:14.628296 28220 raft_consensus.cc:482] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:14.628439 28220 raft_consensus.cc:3037] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.629267 27619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8231124b83946e7bbe6184621bd043d" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" is_pre_election: true
I20240501 08:18:14.629818 27619 raft_consensus.cc:2454] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:14.631029 28220 raft_consensus.cc:504] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.631539 28220 leader_election.cc:288] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Requested vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:14.632050 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8231124b83946e7bbe6184621bd043d" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:14.632212 27619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8231124b83946e7bbe6184621bd043d" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:14.632266 27487 raft_consensus.cc:3037] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.632404 27619 raft_consensus.cc:3037] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.635237 27619 raft_consensus.cc:2454] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:14.635237 27487 raft_consensus.cc:2454] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:14.635637 27683 leader_election.cc:302] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39dff53c8344d3af0bf354b31998eb, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:14.635876 28220 raft_consensus.cc:2781] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:14.636598 28220 raft_consensus.cc:686] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Becoming Leader. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Running, Role: LEADER
I20240501 08:18:14.637095 28220 consensus_queue.cc:237] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.641470 27345 catalog_manager.cc:5484] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:14.714973 28227 raft_consensus.cc:482] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:14.715224 28227 raft_consensus.cc:504] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.717026 28227 leader_election.cc:288] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:14.723969 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41a43a3a85048669f6718a4be9aa382" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045" is_pre_election: true
I20240501 08:18:14.724241 27487 raft_consensus.cc:2454] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 0.
I20240501 08:18:14.724195 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41a43a3a85048669f6718a4be9aa382" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb" is_pre_election: true
I20240501 08:18:14.724571 27750 raft_consensus.cc:2454] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 0.
I20240501 08:18:14.724694 27554 leader_election.cc:302] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:14.725042 28227 raft_consensus.cc:2781] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:14.725165 28227 raft_consensus.cc:482] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:14.725270 28227 raft_consensus.cc:3037] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.727125 28227 raft_consensus.cc:504] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.727661 28227 leader_election.cc:288] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 election: Requested vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:14.728062 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41a43a3a85048669f6718a4be9aa382" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:14.728173 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e41a43a3a85048669f6718a4be9aa382" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
I20240501 08:18:14.728281 27487 raft_consensus.cc:3037] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.728363 27750 raft_consensus.cc:3037] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.730130 27487 raft_consensus.cc:2454] T e41a43a3a85048669f6718a4be9aa382 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 1.
I20240501 08:18:14.730448 27750 raft_consensus.cc:2454] T e41a43a3a85048669f6718a4be9aa382 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 1.
I20240501 08:18:14.730496 27554 leader_election.cc:302] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:14.730755 28227 raft_consensus.cc:2781] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:14.731189 28227 raft_consensus.cc:686] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Becoming Leader. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Running, Role: LEADER
I20240501 08:18:14.731634 28227 consensus_queue.cc:237] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.735685 27345 catalog_manager.cc:5484] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: term changed from 0 to 1, leader changed from <none> to dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130). New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:14.745991 27796 tablet.cc:2369] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:18:14.787366 27665 tablet.cc:2369] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:18:14.841740 28220 raft_consensus.cc:482] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:14.842010 28220 raft_consensus.cc:504] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.843135 28220 leader_election.cc:288] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:14.843641 27619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6e4739226a4fc1a8270753768ce569" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" is_pre_election: true
I20240501 08:18:14.843683 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6e4739226a4fc1a8270753768ce569" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045" is_pre_election: true
I20240501 08:18:14.843907 27619 raft_consensus.cc:2454] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:14.843972 27487 raft_consensus.cc:2454] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:14.844305 27683 leader_election.cc:302] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39dff53c8344d3af0bf354b31998eb, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:14.844586 28220 raft_consensus.cc:2781] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:14.844666 28220 raft_consensus.cc:482] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:14.844777 28220 raft_consensus.cc:3037] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.846529 28220 raft_consensus.cc:504] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.847092 28220 leader_election.cc:288] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Requested vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:14.847436 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6e4739226a4fc1a8270753768ce569" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:14.847513 27619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6e4739226a4fc1a8270753768ce569" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:14.847600 27487 raft_consensus.cc:3037] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.847716 27619 raft_consensus.cc:3037] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:14.849525 27487 raft_consensus.cc:2454] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:14.849610 27619 raft_consensus.cc:2454] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:14.849912 27685 leader_election.cc:302] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb; no voters: 
I20240501 08:18:14.850153 28220 raft_consensus.cc:2781] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:14.850301 28220 raft_consensus.cc:686] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Becoming Leader. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Running, Role: LEADER
I20240501 08:18:14.850543 28220 consensus_queue.cc:237] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:14.853050 27345 catalog_manager.cc:5484] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:15.100371 28220 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:15.107671 28220 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:18:15.203524 28238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:15.203812 28238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:15.210975 28238 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:18:15.260682 28220 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:15.266350 28231 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:15.297562 28227 consensus_queue.cc:1035] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:15.304219 28227 consensus_queue.cc:1035] T e41a43a3a85048669f6718a4be9aa382 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 213.884 ms
  time per row: 0.106942 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:18:16.491753 27344 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60802:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 08:18:16.492480 27344 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:18:16.500761 27465 tablet_service.cc:1467] Processing CreateTablet for tablet 1b27438ef0d3481ca2f18ca39086af08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.501025 27598 tablet_service.cc:1467] Processing CreateTablet for tablet 1b27438ef0d3481ca2f18ca39086af08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.501238 27465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b27438ef0d3481ca2f18ca39086af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.501454 27598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b27438ef0d3481ca2f18ca39086af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.501507 27729 tablet_service.cc:1467] Processing CreateTablet for tablet 1b27438ef0d3481ca2f18ca39086af08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.501926 27729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b27438ef0d3481ca2f18ca39086af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.502501 27599 tablet_service.cc:1467] Processing CreateTablet for tablet dbc3f3548ad74a74855fd55afc53b1b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.502852 27599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc3f3548ad74a74855fd55afc53b1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.505226 27467 tablet_service.cc:1467] Processing CreateTablet for tablet dbc3f3548ad74a74855fd55afc53b1b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.505602 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc3f3548ad74a74855fd55afc53b1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.508280 27597 tablet_service.cc:1467] Processing CreateTablet for tablet 86ccc09027574ba9b014da9428912535 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.508680 27597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ccc09027574ba9b014da9428912535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.512854 27466 tablet_service.cc:1467] Processing CreateTablet for tablet 86ccc09027574ba9b014da9428912535 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.512991 27730 tablet_service.cc:1467] Processing CreateTablet for tablet dbc3f3548ad74a74855fd55afc53b1b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.513017 27728 tablet_service.cc:1467] Processing CreateTablet for tablet 86ccc09027574ba9b014da9428912535 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:16.513208 27466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ccc09027574ba9b014da9428912535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.513375 27730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc3f3548ad74a74855fd55afc53b1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.513561 27728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ccc09027574ba9b014da9428912535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:16.523730 28307 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:16.524268 28305 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:16.525878 28305 tablet_bootstrap.cc:654] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.526880 28307 tablet_bootstrap.cc:654] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.527132 28306 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:16.528306 28305 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:16.528467 28305 ts_tablet_manager.cc:1367] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:18:16.528663 28306 tablet_bootstrap.cc:654] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.529428 28305 raft_consensus.cc:348] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.529755 28305 raft_consensus.cc:374] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.529881 28305 raft_consensus.cc:729] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.530215 28305 consensus_queue.cc:260] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.532666 28307 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:16.532802 28307 ts_tablet_manager.cc:1367] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240501 08:18:16.533473 28307 raft_consensus.cc:348] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.533676 28307 raft_consensus.cc:374] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.533761 28307 raft_consensus.cc:729] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.534034 28307 consensus_queue.cc:260] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.534622 28307 ts_tablet_manager.cc:1398] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:16.534981 28307 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:16.538569 28305 ts_tablet_manager.cc:1398] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20240501 08:18:16.539067 28306 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:16.539264 28306 ts_tablet_manager.cc:1367] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240501 08:18:16.540136 28307 tablet_bootstrap.cc:654] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.540025 28306 raft_consensus.cc:348] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.540918 28306 raft_consensus.cc:374] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.541075 28306 raft_consensus.cc:729] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.541159 28305 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:16.541373 28306 consensus_queue.cc:260] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.543036 28306 ts_tablet_manager.cc:1398] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240501 08:18:16.543109 28305 tablet_bootstrap.cc:654] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.543373 28306 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:16.545786 28307 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:16.545946 28307 ts_tablet_manager.cc:1367] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.003s
I20240501 08:18:16.546444 28307 raft_consensus.cc:348] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.546562 28306 tablet_bootstrap.cc:654] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.546629 28307 raft_consensus.cc:374] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.546720 28307 raft_consensus.cc:729] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.546959 28307 consensus_queue.cc:260] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.547576 28307 ts_tablet_manager.cc:1398] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:16.547888 28307 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:16.549089 28307 tablet_bootstrap.cc:654] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.550793 28305 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:16.550957 28305 ts_tablet_manager.cc:1367] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240501 08:18:16.551720 28306 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:16.551756 28307 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:16.551877 28307 ts_tablet_manager.cc:1367] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240501 08:18:16.551748 28305 raft_consensus.cc:348] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.551992 28305 raft_consensus.cc:374] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.552024 28306 ts_tablet_manager.cc:1367] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.003s
I20240501 08:18:16.552084 28305 raft_consensus.cc:729] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.552347 28305 consensus_queue.cc:260] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.552587 28307 raft_consensus.cc:348] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.552682 28306 raft_consensus.cc:348] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.552803 28307 raft_consensus.cc:374] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.552798 28305 ts_tablet_manager.cc:1398] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:16.552901 28306 raft_consensus.cc:374] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.552901 28307 raft_consensus.cc:729] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.552973 28306 raft_consensus.cc:729] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.553072 28305 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:16.553162 28307 consensus_queue.cc:260] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.553226 28306 consensus_queue.cc:260] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.553676 28306 ts_tablet_manager.cc:1398] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:18:16.553733 28307 ts_tablet_manager.cc:1398] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:16.554358 28306 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:16.554446 28305 tablet_bootstrap.cc:654] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.555774 28306 tablet_bootstrap.cc:654] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:16.559818 28306 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:16.559935 28306 ts_tablet_manager.cc:1367] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240501 08:18:16.560465 28306 raft_consensus.cc:348] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.560597 28306 raft_consensus.cc:374] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.560679 28306 raft_consensus.cc:729] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.560892 28306 consensus_queue.cc:260] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.561300 28306 ts_tablet_manager.cc:1398] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240501 08:18:16.561825 28305 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:16.561947 28305 ts_tablet_manager.cc:1367] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.005s
I20240501 08:18:16.563313 28305 raft_consensus.cc:348] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.563482 28305 raft_consensus.cc:374] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:16.563558 28305 raft_consensus.cc:729] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:16.563768 28305 consensus_queue.cc:260] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.564491 28305 ts_tablet_manager.cc:1398] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:16.610141 28220 raft_consensus.cc:482] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:16.610337 28220 raft_consensus.cc:504] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.610920 28220 leader_election.cc:288] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:16.611480 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc3f3548ad74a74855fd55afc53b1b2" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" is_pre_election: true
I20240501 08:18:16.611460 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc3f3548ad74a74855fd55afc53b1b2" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045" is_pre_election: true
I20240501 08:18:16.611698 27618 raft_consensus.cc:2454] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:16.611717 27487 raft_consensus.cc:2454] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:16.612231 27683 leader_election.cc:302] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39dff53c8344d3af0bf354b31998eb, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:16.612633 28220 raft_consensus.cc:2781] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:16.612743 28220 raft_consensus.cc:482] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:16.612849 28220 raft_consensus.cc:3037] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.614753 28220 raft_consensus.cc:504] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.615300 28220 leader_election.cc:288] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Requested vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:16.615790 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc3f3548ad74a74855fd55afc53b1b2" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:16.615810 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc3f3548ad74a74855fd55afc53b1b2" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:16.616003 27618 raft_consensus.cc:3037] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.616014 27487 raft_consensus.cc:3037] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.616264 28227 raft_consensus.cc:482] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:16.616439 28227 raft_consensus.cc:504] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.617004 28227 leader_election.cc:288] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:16.617580 27486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ccc09027574ba9b014da9428912535" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045" is_pre_election: true
I20240501 08:18:16.617779 27486 raft_consensus.cc:2454] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 0.
I20240501 08:18:16.617777 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ccc09027574ba9b014da9428912535" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb" is_pre_election: true
I20240501 08:18:16.617980 27750 raft_consensus.cc:2454] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 0.
I20240501 08:18:16.618409 27554 leader_election.cc:302] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39dff53c8344d3af0bf354b31998eb, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:16.618676 28227 raft_consensus.cc:2781] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:16.618791 28227 raft_consensus.cc:482] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:16.618906 27487 raft_consensus.cc:2454] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:16.618932 28227 raft_consensus.cc:3037] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.618955 27618 raft_consensus.cc:2454] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:16.619334 27685 leader_election.cc:302] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb; no voters: 
I20240501 08:18:16.619607 28220 raft_consensus.cc:2781] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:16.620012 28220 raft_consensus.cc:686] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Becoming Leader. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Running, Role: LEADER
I20240501 08:18:16.620365 28220 consensus_queue.cc:237] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.621603 28227 raft_consensus.cc:504] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.622167 28227 leader_election.cc:288] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 election: Requested vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:16.622546 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ccc09027574ba9b014da9428912535" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:16.622736 27487 raft_consensus.cc:3037] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.622751 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ccc09027574ba9b014da9428912535" candidate_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
I20240501 08:18:16.622949 27750 raft_consensus.cc:3037] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.623693 27344 catalog_manager.cc:5484] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:16.624753 27750 raft_consensus.cc:2454] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 1.
I20240501 08:18:16.625170 27554 leader_election.cc:302] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39dff53c8344d3af0bf354b31998eb, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:16.625216 27487 raft_consensus.cc:2454] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc0e586a35fa4370bcd23da85a3a4b0b in term 1.
I20240501 08:18:16.625593 28227 raft_consensus.cc:2781] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:16.626092 28227 raft_consensus.cc:686] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Becoming Leader. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Running, Role: LEADER
I20240501 08:18:16.626386 28227 consensus_queue.cc:237] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.628662 27344 catalog_manager.cc:5484] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: term changed from 0 to 1, leader changed from <none> to dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130). New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:16.807371 28220 raft_consensus.cc:482] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:16.807536 28220 raft_consensus.cc:504] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.808169 28220 leader_election.cc:288] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:16.808609 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045" is_pre_election: true
I20240501 08:18:16.808655 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" is_pre_election: true
I20240501 08:18:16.808866 27487 raft_consensus.cc:2454] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:16.808919 27618 raft_consensus.cc:2454] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:16.809278 27685 leader_election.cc:302] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb; no voters: 
I20240501 08:18:16.809531 28220 raft_consensus.cc:2781] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:16.809664 28220 raft_consensus.cc:482] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:16.809760 28220 raft_consensus.cc:3037] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.811547 28220 raft_consensus.cc:504] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.812039 28220 leader_election.cc:288] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Requested vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:16.812409 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:16.812466 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:16.812623 27487 raft_consensus.cc:3037] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.812636 27618 raft_consensus.cc:3037] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:16.815299 27618 raft_consensus.cc:2454] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:16.815299 27487 raft_consensus.cc:2454] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:16.815734 27683 leader_election.cc:302] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:16.815987 28220 raft_consensus.cc:2781] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:16.816152 28220 raft_consensus.cc:686] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Becoming Leader. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Running, Role: LEADER
I20240501 08:18:16.816427 28220 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:16.819031 27344 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:17.005713 28227 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:17.013659 28320 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:17.080338 28318 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:17.085505 28318 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:18:17.207198 28323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:17.207496 28323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:17.214375 28323 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:18:17.353266 27486 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:18:17.353926 28361 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:18:17.391779 27683 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.130:46823, user_credentials={real_user=slave}} blocked reactor thread for 53220.4us
I20240501 08:18:17.392601 27618 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:18:17.393316 28318 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:17.434471 28368 mvcc.cc:205] Tried to move back new op lower bound from 7022802933084807168 to 7022802930962673664. Current Snapshot: MvccSnapshot[applied={T|T < 7022802933084807168}]
I20240501 08:18:17.437271 28358 mvcc.cc:205] Tried to move back new op lower bound from 7022802933084807168 to 7022802930962673664. Current Snapshot: MvccSnapshot[applied={T|T < 7022802933084807168 or (T in {7022802933084807168})}]
I20240501 08:18:17.442792 28377 mvcc.cc:205] Tried to move back new op lower bound from 7022802933084807168 to 7022802930962673664. Current Snapshot: MvccSnapshot[applied={T|T < 7022802933084807168}]

INSERT report
    rows total: 2000
    time total: 208.932 ms
  time per row: 0.104466 ms
I20240501 08:18:17.584760 27344 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60802:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 08:18:17.585527 27344 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:18:17.593514 27598 tablet_service.cc:1467] Processing CreateTablet for tablet c5f38cb98306472c8da2a92f2ebdce57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.593788 27467 tablet_service.cc:1467] Processing CreateTablet for tablet c5f38cb98306472c8da2a92f2ebdce57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.593991 27598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f38cb98306472c8da2a92f2ebdce57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.594179 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f38cb98306472c8da2a92f2ebdce57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.594856 27597 tablet_service.cc:1467] Processing CreateTablet for tablet 596d171152da4a81a0f9276c07063f9d (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.595222 27597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596d171152da4a81a0f9276c07063f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.596549 27599 tablet_service.cc:1467] Processing CreateTablet for tablet 08392402ccb6425aa87e1be0d435bbe6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.596912 27599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08392402ccb6425aa87e1be0d435bbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.599006 27466 tablet_service.cc:1467] Processing CreateTablet for tablet 596d171152da4a81a0f9276c07063f9d (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.599364 27466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596d171152da4a81a0f9276c07063f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.602025 27728 tablet_service.cc:1467] Processing CreateTablet for tablet 596d171152da4a81a0f9276c07063f9d (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.602401 27728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596d171152da4a81a0f9276c07063f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.602329 27730 tablet_service.cc:1467] Processing CreateTablet for tablet 08392402ccb6425aa87e1be0d435bbe6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.602653 27730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08392402ccb6425aa87e1be0d435bbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.604492 27729 tablet_service.cc:1467] Processing CreateTablet for tablet c5f38cb98306472c8da2a92f2ebdce57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.604846 27729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f38cb98306472c8da2a92f2ebdce57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.608772 27465 tablet_service.cc:1467] Processing CreateTablet for tablet 08392402ccb6425aa87e1be0d435bbe6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:17.609153 27465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08392402ccb6425aa87e1be0d435bbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:17.610476 28386 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:17.611313 28385 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:17.612182 28386 tablet_bootstrap.cc:654] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.613247 28385 tablet_bootstrap.cc:654] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.614794 28387 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:17.617022 28386 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:17.617190 28386 ts_tablet_manager.cc:1367] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.003s
I20240501 08:18:17.617640 28385 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:17.617682 28387 tablet_bootstrap.cc:654] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.617779 28385 ts_tablet_manager.cc:1367] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.003s
I20240501 08:18:17.617866 28386 raft_consensus.cc:348] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.618038 28386 raft_consensus.cc:374] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.618162 28386 raft_consensus.cc:729] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.618384 28386 consensus_queue.cc:260] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.618559 28385 raft_consensus.cc:348] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.618863 28385 raft_consensus.cc:374] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.619011 28385 raft_consensus.cc:729] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.619127 28386 ts_tablet_manager.cc:1398] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:17.619391 28386 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:17.619283 28385 consensus_queue.cc:260] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.619910 28385 ts_tablet_manager.cc:1398] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:17.620831 28386 tablet_bootstrap.cc:654] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.621268 28385 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:17.622792 28385 tablet_bootstrap.cc:654] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.625002 28387 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:17.625156 28387 ts_tablet_manager.cc:1367] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240501 08:18:17.626139 28387 raft_consensus.cc:348] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.626391 28387 raft_consensus.cc:374] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.626497 28387 raft_consensus.cc:729] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.626799 28387 consensus_queue.cc:260] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.627602 28387 ts_tablet_manager.cc:1398] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:17.627979 28387 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:17.629031 28386 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:17.629177 28386 ts_tablet_manager.cc:1367] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.003s
I20240501 08:18:17.629523 28387 tablet_bootstrap.cc:654] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.629791 28386 raft_consensus.cc:348] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.630040 28386 raft_consensus.cc:374] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.630160 28386 raft_consensus.cc:729] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.630461 28386 consensus_queue.cc:260] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.631048 28386 ts_tablet_manager.cc:1398] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:17.631385 28386 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:17.632934 28386 tablet_bootstrap.cc:654] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.634552 28385 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:17.634763 28385 ts_tablet_manager.cc:1367] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.006s
I20240501 08:18:17.634846 28386 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: No bootstrap required, opened a new log
I20240501 08:18:17.635013 28387 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:17.635033 28386 ts_tablet_manager.cc:1367] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240501 08:18:17.635205 28387 ts_tablet_manager.cc:1367] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240501 08:18:17.635739 28311 raft_consensus.cc:482] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:17.635661 28386 raft_consensus.cc:348] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.635696 28385 raft_consensus.cc:348] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.635857 28386 raft_consensus.cc:374] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.635931 28385 raft_consensus.cc:374] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.635948 28386 raft_consensus.cc:729] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.635890 28311 raft_consensus.cc:504] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.636001 28385 raft_consensus.cc:729] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.636204 28386 consensus_queue.cc:260] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.636385 28385 consensus_queue.cc:260] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.636675 28386 ts_tablet_manager.cc:1398] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:18:17.637113 28385 ts_tablet_manager.cc:1398] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:17.637387 28385 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b: Bootstrap starting.
I20240501 08:18:17.637764 28311 leader_election.cc:288] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:17.640055 28385 tablet_bootstrap.cc:654] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.643513 28387 raft_consensus.cc:348] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.643812 28387 raft_consensus.cc:374] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.643976 28387 raft_consensus.cc:729] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.644317 28387 consensus_queue.cc:260] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.645098 28387 ts_tablet_manager.cc:1398] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240501 08:18:17.645484 28387 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045: Bootstrap starting.
I20240501 08:18:17.647270 28387 tablet_bootstrap.cc:654] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:17.650393 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596d171152da4a81a0f9276c07063f9d" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" is_pre_election: true
I20240501 08:18:17.650635 27618 raft_consensus.cc:2454] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 0.
I20240501 08:18:17.650902 28385 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b: No bootstrap required, opened a new log
I20240501 08:18:17.651208 27420 leader_election.cc:302] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:17.651675 28311 raft_consensus.cc:2781] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:17.651790 28311 raft_consensus.cc:482] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:17.651892 28311 raft_consensus.cc:3037] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.652371 28385 ts_tablet_manager.cc:1367] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent bootstrapping tablet: real 0.015s	user 0.001s	sys 0.006s
I20240501 08:18:17.653136 28385 raft_consensus.cc:348] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.653359 28385 raft_consensus.cc:374] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.653463 28385 raft_consensus.cc:729] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e586a35fa4370bcd23da85a3a4b0b, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.653683 28385 consensus_queue.cc:260] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.654213 28385 ts_tablet_manager.cc:1398] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240501 08:18:17.655119 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596d171152da4a81a0f9276c07063f9d" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb" is_pre_election: true
I20240501 08:18:17.654819 28311 raft_consensus.cc:504] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.655403 27750 raft_consensus.cc:2454] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 0.
I20240501 08:18:17.655951 28311 leader_election.cc:288] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 election: Requested vote from peers dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:17.656708 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596d171152da4a81a0f9276c07063f9d" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:17.656785 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596d171152da4a81a0f9276c07063f9d" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
I20240501 08:18:17.656937 27750 raft_consensus.cc:3037] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.656955 27618 raft_consensus.cc:3037] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.658794 27750 raft_consensus.cc:2454] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 1.
I20240501 08:18:17.658960 27618 raft_consensus.cc:2454] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 1.
I20240501 08:18:17.659205 27422 leader_election.cc:302] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb; no voters: 
I20240501 08:18:17.659497 28311 raft_consensus.cc:2781] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:17.659905 28387 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045: No bootstrap required, opened a new log
I20240501 08:18:17.660009 28311 raft_consensus.cc:686] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Becoming Leader. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Running, Role: LEADER
I20240501 08:18:17.660082 28387 ts_tablet_manager.cc:1367] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.008s
I20240501 08:18:17.660492 28311 consensus_queue.cc:237] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.660810 28387 raft_consensus.cc:348] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.660953 28387 raft_consensus.cc:374] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:17.661046 28387 raft_consensus.cc:729] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Initialized, Role: FOLLOWER
I20240501 08:18:17.661237 28387 consensus_queue.cc:260] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.661722 28387 ts_tablet_manager.cc:1398] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:18:17.663596 27339 catalog_manager.cc:5484] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64da46caf0204911aba9aba77e5ba045 (127.2.248.129). New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:17.680930 28318 raft_consensus.cc:482] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:17.681128 28318 raft_consensus.cc:504] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.681878 28318 leader_election.cc:288] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823), 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543)
I20240501 08:18:17.682343 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f38cb98306472c8da2a92f2ebdce57" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" is_pre_election: true
I20240501 08:18:17.682447 27486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f38cb98306472c8da2a92f2ebdce57" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045" is_pre_election: true
I20240501 08:18:17.682582 27618 raft_consensus.cc:2454] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:17.682677 27486 raft_consensus.cc:2454] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 0.
I20240501 08:18:17.683049 27683 leader_election.cc:302] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39dff53c8344d3af0bf354b31998eb, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:17.683338 28318 raft_consensus.cc:2781] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:17.683562 28318 raft_consensus.cc:482] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:17.683733 28318 raft_consensus.cc:3037] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.685791 28318 raft_consensus.cc:504] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.686285 28311 raft_consensus.cc:482] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:17.686477 28311 raft_consensus.cc:504] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.686592 28318 leader_election.cc:288] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Requested vote from peers dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823), 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543)
I20240501 08:18:17.686829 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f38cb98306472c8da2a92f2ebdce57" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:17.686846 27486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f38cb98306472c8da2a92f2ebdce57" candidate_uuid: "8b39dff53c8344d3af0bf354b31998eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:17.687022 27618 raft_consensus.cc:3037] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.687040 27486 raft_consensus.cc:3037] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.687197 28311 leader_election.cc:288] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:17.687754 27619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08392402ccb6425aa87e1be0d435bbe6" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" is_pre_election: true
I20240501 08:18:17.687966 27619 raft_consensus.cc:2454] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 0.
I20240501 08:18:17.688139 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08392402ccb6425aa87e1be0d435bbe6" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb" is_pre_election: true
I20240501 08:18:17.688362 27750 raft_consensus.cc:2454] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 0.
I20240501 08:18:17.688598 27420 leader_election.cc:302] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:17.689067 28311 raft_consensus.cc:2781] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:17.689164 28311 raft_consensus.cc:482] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:17.689263 28311 raft_consensus.cc:3037] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.689844 27618 raft_consensus.cc:2454] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:17.689863 27486 raft_consensus.cc:2454] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b39dff53c8344d3af0bf354b31998eb in term 1.
I20240501 08:18:17.690241 27685 leader_election.cc:302] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb; no voters: 
I20240501 08:18:17.690541 28318 raft_consensus.cc:2781] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:17.690835 28318 raft_consensus.cc:686] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Becoming Leader. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Running, Role: LEADER
I20240501 08:18:17.691159 28318 consensus_queue.cc:237] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.691689 28311 raft_consensus.cc:504] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.692374 28311 leader_election.cc:288] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 election: Requested vote from peers dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:17.692574 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08392402ccb6425aa87e1be0d435bbe6" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:17.692631 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08392402ccb6425aa87e1be0d435bbe6" candidate_uuid: "64da46caf0204911aba9aba77e5ba045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
I20240501 08:18:17.692822 27750 raft_consensus.cc:3037] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.692813 27618 raft_consensus.cc:3037] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:17.694576 27750 raft_consensus.cc:2454] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 1.
I20240501 08:18:17.694635 27618 raft_consensus.cc:2454] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64da46caf0204911aba9aba77e5ba045 in term 1.
I20240501 08:18:17.694916 27422 leader_election.cc:302] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 8b39dff53c8344d3af0bf354b31998eb; no voters: 
I20240501 08:18:17.695164 28311 raft_consensus.cc:2781] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:17.695374 28311 raft_consensus.cc:686] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Becoming Leader. State: Replica: 64da46caf0204911aba9aba77e5ba045, State: Running, Role: LEADER
I20240501 08:18:17.695655 28311 consensus_queue.cc:237] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } }
I20240501 08:18:17.696357 27339 catalog_manager.cc:5484] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131). New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:17.698354 27344 catalog_manager.cc:5484] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64da46caf0204911aba9aba77e5ba045 (127.2.248.129). New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:18.027803 28403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:18.028091 28403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:18.035418 28403 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:18:18.049729 28311 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:18.055773 28416 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 08:18:18.072057 28362 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:18.075440 28399 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:18.077610 28362 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:18.086839 28399 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 213.095 ms
  time per row: 0.106548 ms
I20240501 08:18:18.414709  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:34905
--local_ip_for_outbound_sockets=127.2.248.132
--tserver_master_addrs=127.2.248.190:45677
--webserver_port=39633
--webserver_interface=127.2.248.132
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:18:18.748884 28462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:18.749197 28462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:18.749382 28462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:18.756716 28462 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:18.756827 28462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:18.756940 28462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:18:18.765571 28462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:34905
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=39633
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:18.766736 28462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:18.768389 28462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:18.780452 28470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:18.781018 28467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:18.781045 28468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:18.781625 28462 server_base.cc:1034] running on GCE node
I20240501 08:18:18.782223 28462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:18.783128 28462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:18.784307 28462 hybrid_clock.cc:648] HybridClock initialized: now 1714551498784255 us; error 51 us; skew 500 ppm
I20240501 08:18:18.784615 28462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:18.786883 28462 webserver.cc:460] Webserver started at http://127.2.248.132:39633/ using document root <none> and password file <none>
I20240501 08:18:18.787554 28462 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:18.787674 28462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:18.792838 28462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240501 08:18:18.795693 28476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:18.796931 28462 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:18:18.797078 28462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9a400de32dd143fbbf6485d0ddc1dba7"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:18.797835 28462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:18.813509 28462 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:18.814217 28462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:18.814469 28462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:18.815285 28462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:18.816673 28462 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:18.816773 28462 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:18.816890 28462 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:18.816946 28462 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:18.847095 28462 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:34905
I20240501 08:18:18.847129 28588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:34905 every 8 connection(s)
I20240501 08:18:18.848862 28462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:18:18.853264  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 28462
I20240501 08:18:18.853804  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:42453
--local_ip_for_outbound_sockets=127.2.248.133
--tserver_master_addrs=127.2.248.190:45677
--webserver_port=46265
--webserver_interface=127.2.248.133
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:18.861541 28589 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:18.861850 28589 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:18.862788 28589 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:18.864322 27341 ts_manager.cc:194] Re-registered known tserver with Master: 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132:34905)
I20240501 08:18:18.865150 27341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:36053
W20240501 08:18:19.188221 28593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:19.188521 28593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:19.188606 28593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:19.195586 28593 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:19.195698 28593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:19.195838 28593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:18:19.204609 28593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:42453
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=46265
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:19.205888 28593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:19.207487 28593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:19.220085 28598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:19.220525 28599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:19.220989 28601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:19.221251 28593 server_base.cc:1034] running on GCE node
I20240501 08:18:19.221873 28593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:19.222798 28593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:19.224058 28593 hybrid_clock.cc:648] HybridClock initialized: now 1714551499224008 us; error 70 us; skew 500 ppm
I20240501 08:18:19.224417 28593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:19.226943 28593 webserver.cc:460] Webserver started at http://127.2.248.133:46265/ using document root <none> and password file <none>
I20240501 08:18:19.227631 28593 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:19.227752 28593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:19.233285 28593 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240501 08:18:19.235976 28607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:19.237247 28593 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:18:19.237401 28593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b755b190bbfc4bc0a08c28c7263c94cd"
format_stamp: "Formatted at 2024-05-01 08:18:09 on dist-test-slave-0pdg"
I20240501 08:18:19.238131 28593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:19.265931 28593 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:19.266682 28593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:19.266965 28593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:19.267855 28593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:19.269536 28593 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:19.269625 28593 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:19.269757 28593 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:19.269831 28593 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:19.301565 28593 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:42453
I20240501 08:18:19.301620 28719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:42453 every 8 connection(s)
I20240501 08:18:19.303433 28593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:18:19.304981  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 28593
I20240501 08:18:19.305547  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.134:41033
--local_ip_for_outbound_sockets=127.2.248.134
--tserver_master_addrs=127.2.248.190:45677
--webserver_port=39295
--webserver_interface=127.2.248.134
--builtin_ntp_servers=127.2.248.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:19.320611 28720 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:19.321004 28720 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:19.322010 28720 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:19.323596 27341 ts_manager.cc:194] Re-registered known tserver with Master: b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133:42453)
I20240501 08:18:19.324393 27341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:49953
W20240501 08:18:19.643157 28723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:19.643429 28723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:19.643564 28723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:19.651546 28723 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:19.651660 28723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:19.651827 28723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.134
I20240501 08:18:19.660620 28723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.134:41033
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.2.248.134
--webserver_port=39295
--tserver_master_addrs=127.2.248.190:45677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.134
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:19.661926 28723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:19.663655 28723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:19.675752 28729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:19.676240 28728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:19.676641 28731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:19.677225 28723 server_base.cc:1034] running on GCE node
I20240501 08:18:19.677803 28723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:19.678740 28723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:19.679952 28723 hybrid_clock.cc:648] HybridClock initialized: now 1714551499679905 us; error 38 us; skew 500 ppm
I20240501 08:18:19.680296 28723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:19.682735 28723 webserver.cc:460] Webserver started at http://127.2.248.134:39295/ using document root <none> and password file <none>
I20240501 08:18:19.683424 28723 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:19.683545 28723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:19.688800 28723 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:18:19.691578 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:19.692983 28723 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:19.693120 28723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "40c65a4181944bf69feec459e131a224"
format_stamp: "Formatted at 2024-05-01 08:18:10 on dist-test-slave-0pdg"
I20240501 08:18:19.693915 28723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:19.710198 28723 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:19.710951 28723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:19.711194 28723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:19.712093 28723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:19.713501 28723 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:19.713563 28723 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:19.713652 28723 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:19.713696 28723 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:19.756443 28723 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.134:41033
I20240501 08:18:19.756445 28849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.134:41033 every 8 connection(s)
I20240501 08:18:19.757869 28723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:18:19.759037  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 28723
I20240501 08:18:19.784689 28850 heartbeater.cc:347] Connected to a master server at 127.2.248.190:45677
I20240501 08:18:19.785066 28850 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:19.786128 28850 heartbeater.cc:511] Master 127.2.248.190:45677 requested a full tablet report, sending...
I20240501 08:18:19.787746 27341 ts_manager.cc:194] Re-registered known tserver with Master: 40c65a4181944bf69feec459e131a224 (127.2.248.134:41033)
I20240501 08:18:19.788637 27341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.134:57487
I20240501 08:18:19.867911 28589 heartbeater.cc:503] Master 127.2.248.190:45677 was elected leader, sending a full tablet report...
I20240501 08:18:20.240588 28524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.246891 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.257192 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.282722 27728 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:20.283911 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.307722 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.327168 28720 heartbeater.cc:503] Master 127.2.248.190:45677 was elected leader, sending a full tablet report...
I20240501 08:18:20.457235 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.463384 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.474572 27728 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:20.476608 28524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.510825 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.540052 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.681999 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.689260 27728 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:20.693360 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.697160 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.703325 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.703886 28524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.791712 28850 heartbeater.cc:503] Master 127.2.248.190:45677 was elected leader, sending a full tablet report...
I20240501 08:18:20.880272 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.890620 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.892796 28524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.897603 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.899892 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:20.906378 27728 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:20.983892 27750 consensus_queue.cc:237] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:20.986198 27617 raft_consensus.cc:1261] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:20.986438 27486 raft_consensus.cc:1261] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:20.986968 29000 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:20.987293 29001 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:18:20.993432 27686 consensus_peers.cc:485] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb -> Peer 40c65a4181944bf69feec459e131a224 (127.2.248.134:41033): Couldn't send request to peer 40c65a4181944bf69feec459e131a224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5f38cb98306472c8da2a92f2ebdce57. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:20.992753 29000 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:20.993117 27486 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:20.997023 27617 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:21.002066 27341 catalog_manager.cc:5484] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:21.020120 27617 consensus_queue.cc:237] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:21.022619 27486 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:21.022935 27750 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:21.023393 29015 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:21.023679 28320 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:21.026849 28320 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:21.030072 27750 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:21.031368 27486 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:21.033129 27342 catalog_manager.cc:5484] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:21.037638 27553 consensus_peers.cc:485] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b -> Peer 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132:34905): Couldn't send request to peer 9a400de32dd143fbbf6485d0ddc1dba7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86ccc09027574ba9b014da9428912535. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:21.062047 27750 consensus_queue.cc:237] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:21.064277 27617 raft_consensus.cc:1261] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:21.064380 27486 raft_consensus.cc:1261] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:21.064954 29000 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:21.065527 28362 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:18:21.073017 27686 consensus_peers.cc:485] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb -> Peer b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133:42453): Couldn't send request to peer b755b190bbfc4bc0a08c28c7263c94cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f6e4739226a4fc1a8270753768ce569. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:21.078207 29001 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:21.080715 27486 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:21.084057 27341 catalog_manager.cc:5484] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:21.087759 27617 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:21.106115 27750 consensus_queue.cc:237] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:21.108098 27617 raft_consensus.cc:1261] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:21.108160 27486 raft_consensus.cc:1261] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:18:21.108651 27686 consensus_peers.cc:485] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb -> Peer 40c65a4181944bf69feec459e131a224 (127.2.248.134:41033): Couldn't send request to peer 40c65a4181944bf69feec459e131a224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8231124b83946e7bbe6184621bd043d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:21.108714 29001 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:21.109006 29000 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:21.111779 29001 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:21.112473 27617 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:21.112540 27486 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:21.116557 27339 catalog_manager.cc:5484] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:21.129314 27486 consensus_queue.cc:237] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:21.132536 27750 raft_consensus.cc:1261] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:21.132076 27617 raft_consensus.cc:1261] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:21.133265 28311 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:21.134400 28311 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:21.138183 28311 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:21.140859 27750 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:21.145277 27618 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:21.150343 27342 catalog_manager.cc:5484] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: config changed from index -1 to 4, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
W20240501 08:18:21.154861 27423 consensus_peers.cc:485] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 -> Peer b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133:42453): Couldn't send request to peer b755b190bbfc4bc0a08c28c7263c94cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 596d171152da4a81a0f9276c07063f9d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:21.492820 29099 ts_tablet_manager.cc:902] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: Initiating tablet copy from peer 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:21.494457 29099 tablet_copy_client.cc:308] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:38383
I20240501 08:18:21.500394 27770 tablet_copy_service.cc:140] P 8b39dff53c8344d3af0bf354b31998eb: Received BeginTabletCopySession request for tablet c5f38cb98306472c8da2a92f2ebdce57 from peer 40c65a4181944bf69feec459e131a224 ({username='slave'} at 127.2.248.134:39835)
I20240501 08:18:21.500631 27770 tablet_copy_service.cc:161] P 8b39dff53c8344d3af0bf354b31998eb: Beginning new tablet copy session on tablet c5f38cb98306472c8da2a92f2ebdce57 from peer 40c65a4181944bf69feec459e131a224 at {username='slave'} at 127.2.248.134:39835: session id = 40c65a4181944bf69feec459e131a224-c5f38cb98306472c8da2a92f2ebdce57
I20240501 08:18:21.506817 27770 tablet_copy_source_session.cc:205] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:21.511497 29099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f38cb98306472c8da2a92f2ebdce57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:21.513900 29105 ts_tablet_manager.cc:902] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Initiating tablet copy from peer dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:21.515795 29105 tablet_copy_client.cc:308] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46823
I20240501 08:18:21.522266 29099 tablet_copy_client.cc:783] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:21.522562 29099 tablet_copy_client.cc:647] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:21.522491 27639 tablet_copy_service.cc:140] P dc0e586a35fa4370bcd23da85a3a4b0b: Received BeginTabletCopySession request for tablet 86ccc09027574ba9b014da9428912535 from peer 9a400de32dd143fbbf6485d0ddc1dba7 ({username='slave'} at 127.2.248.132:45061)
I20240501 08:18:21.522774 27639 tablet_copy_service.cc:161] P dc0e586a35fa4370bcd23da85a3a4b0b: Beginning new tablet copy session on tablet 86ccc09027574ba9b014da9428912535 from peer 9a400de32dd143fbbf6485d0ddc1dba7 at {username='slave'} at 127.2.248.132:45061: session id = 9a400de32dd143fbbf6485d0ddc1dba7-86ccc09027574ba9b014da9428912535
I20240501 08:18:21.524963 29099 tablet_copy_client.cc:523] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:21.525796 27639 tablet_copy_source_session.cc:205] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:21.527915 29099 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: Bootstrap starting.
I20240501 08:18:21.530354 29105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ccc09027574ba9b014da9428912535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:21.538882 29099 log.cc:824] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:21.540633 29105 tablet_copy_client.cc:783] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:21.540969 29105 tablet_copy_client.cc:647] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:21.543342 29105 tablet_copy_client.cc:523] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:21.546399 29105 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap starting.
I20240501 08:18:21.555967 29105 log.cc:824] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:21.576745 29113 ts_tablet_manager.cc:902] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: Initiating tablet copy from peer 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543)
I20240501 08:18:21.593637 29099 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:21.594298 29099 tablet_bootstrap.cc:492] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: Bootstrap complete.
I20240501 08:18:21.594798 29099 ts_tablet_manager.cc:1367] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: Time spent bootstrapping tablet: real 0.067s	user 0.067s	sys 0.000s
I20240501 08:18:21.603392 29116 ts_tablet_manager.cc:902] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: Initiating tablet copy from peer 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:21.603658 29099 raft_consensus.cc:348] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:21.604285 29099 raft_consensus.cc:729] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40c65a4181944bf69feec459e131a224, State: Initialized, Role: LEARNER
I20240501 08:18:21.604964 29099 consensus_queue.cc:260] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:21.613736 29099 ts_tablet_manager.cc:1398] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224: Time spent starting tablet: real 0.019s	user 0.012s	sys 0.004s
I20240501 08:18:21.615052 27770 tablet_copy_service.cc:342] P 8b39dff53c8344d3af0bf354b31998eb: Request end of tablet copy session 40c65a4181944bf69feec459e131a224-c5f38cb98306472c8da2a92f2ebdce57 received from {username='slave'} at 127.2.248.134:39835
I20240501 08:18:21.615242 27770 tablet_copy_service.cc:433] P 8b39dff53c8344d3af0bf354b31998eb: ending tablet copy session 40c65a4181944bf69feec459e131a224-c5f38cb98306472c8da2a92f2ebdce57 on tablet c5f38cb98306472c8da2a92f2ebdce57 with peer 40c65a4181944bf69feec459e131a224
I20240501 08:18:21.618288 29116 tablet_copy_client.cc:308] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:38383
I20240501 08:18:21.618981 27770 tablet_copy_service.cc:140] P 8b39dff53c8344d3af0bf354b31998eb: Received BeginTabletCopySession request for tablet a8231124b83946e7bbe6184621bd043d from peer 40c65a4181944bf69feec459e131a224 ({username='slave'} at 127.2.248.134:39835)
I20240501 08:18:21.619045 29113 tablet_copy_client.cc:308] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44543
I20240501 08:18:21.619170 27770 tablet_copy_service.cc:161] P 8b39dff53c8344d3af0bf354b31998eb: Beginning new tablet copy session on tablet a8231124b83946e7bbe6184621bd043d from peer 40c65a4181944bf69feec459e131a224 at {username='slave'} at 127.2.248.134:39835: session id = 40c65a4181944bf69feec459e131a224-a8231124b83946e7bbe6184621bd043d
I20240501 08:18:21.622061 27770 tablet_copy_source_session.cc:205] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:21.623245 29116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8231124b83946e7bbe6184621bd043d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:21.628005 29116 tablet_copy_client.cc:783] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:21.628304 29116 tablet_copy_client.cc:647] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:21.629106 29105 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:21.629698 29105 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap complete.
I20240501 08:18:21.630122 29116 tablet_copy_client.cc:523] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:21.630244 29105 ts_tablet_manager.cc:1367] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Time spent bootstrapping tablet: real 0.084s	user 0.065s	sys 0.017s
I20240501 08:18:21.630157 27507 tablet_copy_service.cc:140] P 64da46caf0204911aba9aba77e5ba045: Received BeginTabletCopySession request for tablet 596d171152da4a81a0f9276c07063f9d from peer b755b190bbfc4bc0a08c28c7263c94cd ({username='slave'} at 127.2.248.133:50233)
I20240501 08:18:21.630429 27507 tablet_copy_service.cc:161] P 64da46caf0204911aba9aba77e5ba045: Beginning new tablet copy session on tablet 596d171152da4a81a0f9276c07063f9d from peer b755b190bbfc4bc0a08c28c7263c94cd at {username='slave'} at 127.2.248.133:50233: session id = b755b190bbfc4bc0a08c28c7263c94cd-596d171152da4a81a0f9276c07063f9d
I20240501 08:18:21.633147 27507 tablet_copy_source_session.cc:205] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:21.633468 29116 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: Bootstrap starting.
I20240501 08:18:21.636382 29105 raft_consensus.cc:348] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:21.636845 29105 raft_consensus.cc:729] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a400de32dd143fbbf6485d0ddc1dba7, State: Initialized, Role: LEARNER
I20240501 08:18:21.637477 29113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596d171152da4a81a0f9276c07063f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:21.637403 29105 consensus_queue.cc:260] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:21.639278 29105 ts_tablet_manager.cc:1398] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.004s
I20240501 08:18:21.640588 27639 tablet_copy_service.cc:342] P dc0e586a35fa4370bcd23da85a3a4b0b: Request end of tablet copy session 9a400de32dd143fbbf6485d0ddc1dba7-86ccc09027574ba9b014da9428912535 received from {username='slave'} at 127.2.248.132:45061
I20240501 08:18:21.640789 27639 tablet_copy_service.cc:433] P dc0e586a35fa4370bcd23da85a3a4b0b: ending tablet copy session 9a400de32dd143fbbf6485d0ddc1dba7-86ccc09027574ba9b014da9428912535 on tablet 86ccc09027574ba9b014da9428912535 with peer 9a400de32dd143fbbf6485d0ddc1dba7
I20240501 08:18:21.651578 29113 tablet_copy_client.cc:783] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:21.651928 29113 tablet_copy_client.cc:647] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:21.654647 29113 tablet_copy_client.cc:523] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:21.658784 29113 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap starting.
I20240501 08:18:21.669358 29113 log.cc:824] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:21.696593 29143 ts_tablet_manager.cc:902] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: Initiating tablet copy from peer 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:21.697407 29116 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:21.698709 29116 tablet_bootstrap.cc:492] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: Bootstrap complete.
I20240501 08:18:21.699167 29116 ts_tablet_manager.cc:1367] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: Time spent bootstrapping tablet: real 0.066s	user 0.059s	sys 0.005s
I20240501 08:18:21.702455 29116 raft_consensus.cc:348] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:21.702845 29116 raft_consensus.cc:729] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40c65a4181944bf69feec459e131a224, State: Initialized, Role: LEARNER
I20240501 08:18:21.703181 29116 consensus_queue.cc:260] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:21.704173 29116 ts_tablet_manager.cc:1398] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:18:21.704916 27770 tablet_copy_service.cc:342] P 8b39dff53c8344d3af0bf354b31998eb: Request end of tablet copy session 40c65a4181944bf69feec459e131a224-a8231124b83946e7bbe6184621bd043d received from {username='slave'} at 127.2.248.134:39835
I20240501 08:18:21.705050 27770 tablet_copy_service.cc:433] P 8b39dff53c8344d3af0bf354b31998eb: ending tablet copy session 40c65a4181944bf69feec459e131a224-a8231124b83946e7bbe6184621bd043d on tablet a8231124b83946e7bbe6184621bd043d with peer 40c65a4181944bf69feec459e131a224
I20240501 08:18:21.711686 29143 tablet_copy_client.cc:308] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:38383
I20240501 08:18:21.719383 29113 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:21.719964 29113 tablet_bootstrap.cc:492] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap complete.
I20240501 08:18:21.720443 29113 ts_tablet_manager.cc:1367] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: Time spent bootstrapping tablet: real 0.062s	user 0.059s	sys 0.003s
I20240501 08:18:21.720500 27770 tablet_copy_service.cc:140] P 8b39dff53c8344d3af0bf354b31998eb: Received BeginTabletCopySession request for tablet 7f6e4739226a4fc1a8270753768ce569 from peer b755b190bbfc4bc0a08c28c7263c94cd ({username='slave'} at 127.2.248.133:44821)
I20240501 08:18:21.720666 27770 tablet_copy_service.cc:161] P 8b39dff53c8344d3af0bf354b31998eb: Beginning new tablet copy session on tablet 7f6e4739226a4fc1a8270753768ce569 from peer b755b190bbfc4bc0a08c28c7263c94cd at {username='slave'} at 127.2.248.133:44821: session id = b755b190bbfc4bc0a08c28c7263c94cd-7f6e4739226a4fc1a8270753768ce569
I20240501 08:18:21.723388 27770 tablet_copy_source_session.cc:205] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:21.724551 29143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6e4739226a4fc1a8270753768ce569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:21.726090 29113 raft_consensus.cc:348] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:21.726589 29113 raft_consensus.cc:729] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b755b190bbfc4bc0a08c28c7263c94cd, State: Initialized, Role: LEARNER
I20240501 08:18:21.727233 29113 consensus_queue.cc:260] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:21.729684 29113 ts_tablet_manager.cc:1398] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20240501 08:18:21.730823 27507 tablet_copy_service.cc:342] P 64da46caf0204911aba9aba77e5ba045: Request end of tablet copy session b755b190bbfc4bc0a08c28c7263c94cd-596d171152da4a81a0f9276c07063f9d received from {username='slave'} at 127.2.248.133:50233
I20240501 08:18:21.731007 27507 tablet_copy_service.cc:433] P 64da46caf0204911aba9aba77e5ba045: ending tablet copy session b755b190bbfc4bc0a08c28c7263c94cd-596d171152da4a81a0f9276c07063f9d on tablet 596d171152da4a81a0f9276c07063f9d with peer b755b190bbfc4bc0a08c28c7263c94cd
I20240501 08:18:21.732479 29143 tablet_copy_client.cc:783] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:21.732733 29143 tablet_copy_client.cc:647] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:21.734237 29143 tablet_copy_client.cc:523] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:21.736474 29143 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap starting.
I20240501 08:18:21.770406 29143 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:21.770846 29143 tablet_bootstrap.cc:492] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap complete.
I20240501 08:18:21.771137 29143 ts_tablet_manager.cc:1367] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: Time spent bootstrapping tablet: real 0.035s	user 0.032s	sys 0.000s
I20240501 08:18:21.772082 29143 raft_consensus.cc:348] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:21.772403 29143 raft_consensus.cc:729] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b755b190bbfc4bc0a08c28c7263c94cd, State: Initialized, Role: LEARNER
I20240501 08:18:21.772619 29143 consensus_queue.cc:260] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:21.773191 29143 ts_tablet_manager.cc:1398] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240501 08:18:21.774042 27770 tablet_copy_service.cc:342] P 8b39dff53c8344d3af0bf354b31998eb: Request end of tablet copy session b755b190bbfc4bc0a08c28c7263c94cd-7f6e4739226a4fc1a8270753768ce569 received from {username='slave'} at 127.2.248.133:44821
I20240501 08:18:21.774197 27770 tablet_copy_service.cc:433] P 8b39dff53c8344d3af0bf354b31998eb: ending tablet copy session b755b190bbfc4bc0a08c28c7263c94cd-7f6e4739226a4fc1a8270753768ce569 on tablet 7f6e4739226a4fc1a8270753768ce569 with peer b755b190bbfc4bc0a08c28c7263c94cd
I20240501 08:18:21.941886 28544 raft_consensus.cc:1203] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:21.965360 28805 raft_consensus.cc:1203] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:22.055802 28675 raft_consensus.cc:1203] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:22.092839 28805 raft_consensus.cc:1203] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:22.228668 28675 raft_consensus.cc:1203] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:22.458318 29231 raft_consensus.cc:1050] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: attempting to promote NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 to VOTER
I20240501 08:18:22.459451 29231 consensus_queue.cc:237] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:22.462374 28544 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.463008 27750 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.463074 29166 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.463762 29166 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.475847 29231 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.476329 27486 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.476585 27750 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.476915 29166 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.481463 27486 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.482717 27342 catalog_manager.cc:5484] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.482439 28544 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.512670 29000 raft_consensus.cc:1050] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb: attempting to promote NON_VOTER 40c65a4181944bf69feec459e131a224 to VOTER
I20240501 08:18:22.513449 29000 consensus_queue.cc:237] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } }
I20240501 08:18:22.515688 27617 raft_consensus.cc:1261] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.515725 28805 raft_consensus.cc:1261] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.515820 27486 raft_consensus.cc:1261] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.516270 29000 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.516611 29253 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.517004 29183 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.521191 29183 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.521919 27617 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.522697 28805 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.525257 27486 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.526921 27342 catalog_manager.cc:5484] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:22.531848 27750 consensus_queue.cc:237] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } }
I20240501 08:18:22.532219 29000 raft_consensus.cc:1050] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb: attempting to promote NON_VOTER 40c65a4181944bf69feec459e131a224 to VOTER
I20240501 08:18:22.533069 29000 consensus_queue.cc:237] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } }
I20240501 08:18:22.534070 27617 raft_consensus.cc:1261] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.535183 29253 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.535596 27486 raft_consensus.cc:1261] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.536149 28804 raft_consensus.cc:1261] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.536157 29183 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.536857 29000 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.538365 29190 raft_consensus.cc:1050] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045: attempting to promote NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd to VOTER
I20240501 08:18:22.539722 29190 consensus_queue.cc:237] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } }
I20240501 08:18:22.548486 28805 raft_consensus.cc:1261] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.549238 29183 consensus_queue.cc:1035] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.552449 29000 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.553170 28803 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.556828 27332 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8231124b83946e7bbe6184621bd043d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:22.558779 27339 catalog_manager.cc:5484] T a8231124b83946e7bbe6184621bd043d P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:22.563894 28675 raft_consensus.cc:1261] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.565204 27466 tablet_service.cc:1514] Processing DeleteTablet for tablet a8231124b83946e7bbe6184621bd043d with delete_type TABLET_DATA_TOMBSTONED (TS 64da46caf0204911aba9aba77e5ba045 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42448
I20240501 08:18:22.565905 27617 raft_consensus.cc:1261] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.565832 28311 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.567183 27617 raft_consensus.cc:2932] T a8231124b83946e7bbe6184621bd043d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.571542 29253 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.581140 27750 raft_consensus.cc:1261] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.581403 27617 raft_consensus.cc:1261] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.581677 29189 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.583357 28311 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.585637 29262 tablet_replica.cc:330] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: stopping tablet replica
I20240501 08:18:22.589319 29262 raft_consensus.cc:2229] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:22.589762 29262 raft_consensus.cc:2258] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:22.591801 29258 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.592646 27486 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.593436 28803 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.596418 27617 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.596923 29262 ts_tablet_manager.cc:1875] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:22.598067 27341 catalog_manager.cc:5484] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:22.605279 27749 consensus_queue.cc:237] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } }
I20240501 08:18:22.607896 29190 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.608729 27619 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.612336 28805 raft_consensus.cc:1261] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.613406 29258 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.615054 27339 catalog_manager.cc:5484] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 reported cstate change: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:22.616261 27617 raft_consensus.cc:1261] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.617060 29000 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.617754 29253 consensus_queue.cc:1035] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.617229 28675 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.619041 27617 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.620782 27332 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5f38cb98306472c8da2a92f2ebdce57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:22.623795 27341 catalog_manager.cc:5484] T c5f38cb98306472c8da2a92f2ebdce57 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:22.624572 27750 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.621790 28804 raft_consensus.cc:2932] T c5f38cb98306472c8da2a92f2ebdce57 P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:22.628008 27466 tablet_service.cc:1514] Processing DeleteTablet for tablet c5f38cb98306472c8da2a92f2ebdce57 with delete_type TABLET_DATA_TOMBSTONED (TS 64da46caf0204911aba9aba77e5ba045 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42448
I20240501 08:18:22.629602 27486 consensus_queue.cc:237] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } }
I20240501 08:18:22.631800 27750 raft_consensus.cc:1261] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.632253 29190 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.632763 28674 raft_consensus.cc:1261] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.634071 29190 consensus_queue.cc:1035] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.635010 29190 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.635360 27750 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.636386 29262 ts_tablet_manager.cc:1888] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:22.636547 29262 log.cc:1196] T a8231124b83946e7bbe6184621bd043d P 64da46caf0204911aba9aba77e5ba045: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/a8231124b83946e7bbe6184621bd043d
I20240501 08:18:22.637454 29262 tablet_replica.cc:330] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: stopping tablet replica
I20240501 08:18:22.637670 29262 raft_consensus.cc:2229] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:22.637569 27332 catalog_manager.cc:4830] TS 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543): tablet a8231124b83946e7bbe6184621bd043d (table rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]) successfully deleted
I20240501 08:18:22.637881 29262 raft_consensus.cc:2258] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:22.638808 29262 ts_tablet_manager.cc:1875] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:22.638801 27332 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 596d171152da4a81a0f9276c07063f9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:22.638986 28674 raft_consensus.cc:2932] T 596d171152da4a81a0f9276c07063f9d P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.641000 27341 catalog_manager.cc:5484] T 596d171152da4a81a0f9276c07063f9d P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.644466 29262 ts_tablet_manager.cc:1888] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:22.644567 29262 log.cc:1196] T c5f38cb98306472c8da2a92f2ebdce57 P 64da46caf0204911aba9aba77e5ba045: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5f38cb98306472c8da2a92f2ebdce57
I20240501 08:18:22.645449 27332 catalog_manager.cc:4830] TS 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543): tablet c5f38cb98306472c8da2a92f2ebdce57 (table rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]) successfully deleted
I20240501 08:18:22.645642 27598 tablet_service.cc:1514] Processing DeleteTablet for tablet 596d171152da4a81a0f9276c07063f9d with delete_type TABLET_DATA_TOMBSTONED (TS dc0e586a35fa4370bcd23da85a3a4b0b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48666
I20240501 08:18:22.646600 29267 tablet_replica.cc:330] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: stopping tablet replica
I20240501 08:18:22.647117 29267 raft_consensus.cc:2229] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:22.647468 29267 raft_consensus.cc:2258] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:22.648875 29267 ts_tablet_manager.cc:1875] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:22.653184 29267 ts_tablet_manager.cc:1888] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:22.653290 29267 log.cc:1196] T 596d171152da4a81a0f9276c07063f9d P dc0e586a35fa4370bcd23da85a3a4b0b: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/596d171152da4a81a0f9276c07063f9d
I20240501 08:18:22.653949 27330 catalog_manager.cc:4830] TS dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823): tablet 596d171152da4a81a0f9276c07063f9d (table rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]) successfully deleted
I20240501 08:18:22.834093 29258 raft_consensus.cc:1050] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb: attempting to promote NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd to VOTER
I20240501 08:18:22.834723 29258 consensus_queue.cc:237] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } }
I20240501 08:18:22.837074 27486 raft_consensus.cc:1261] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.837078 28674 raft_consensus.cc:1261] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.837338 27617 raft_consensus.cc:1261] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:22.837616 29253 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.837905 29000 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.838204 29183 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:22.840931 29183 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.841735 28674 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.844693 27486 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.849947 27617 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.853180 27339 catalog_manager.cc:5484] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index 4 to 5, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:22.857972 27750 consensus_queue.cc:237] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } }
I20240501 08:18:22.859721 27486 raft_consensus.cc:1261] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.859717 28674 raft_consensus.cc:1261] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.860262 29000 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.860514 29183 consensus_queue.cc:1035] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.862776 29253 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.863327 28674 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.863523 27486 raft_consensus.cc:2932] T 7f6e4739226a4fc1a8270753768ce569 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.866186 27332 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f6e4739226a4fc1a8270753768ce569 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:22.867007 27342 catalog_manager.cc:5484] T 7f6e4739226a4fc1a8270753768ce569 P b755b190bbfc4bc0a08c28c7263c94cd reported cstate change: config changed from index 5 to 6, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:22.870325 27598 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f6e4739226a4fc1a8270753768ce569 with delete_type TABLET_DATA_TOMBSTONED (TS dc0e586a35fa4370bcd23da85a3a4b0b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48666
I20240501 08:18:22.870646 29267 tablet_replica.cc:330] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: stopping tablet replica
I20240501 08:18:22.870844 29267 raft_consensus.cc:2229] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:22.871174 29267 raft_consensus.cc:2258] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:22.872614 29267 ts_tablet_manager.cc:1875] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:22.876557 29267 ts_tablet_manager.cc:1888] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:22.876646 29267 log.cc:1196] T 7f6e4739226a4fc1a8270753768ce569 P dc0e586a35fa4370bcd23da85a3a4b0b: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/7f6e4739226a4fc1a8270753768ce569
I20240501 08:18:22.877270 27330 catalog_manager.cc:4830] TS dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823): tablet 7f6e4739226a4fc1a8270753768ce569 (table rebalance_test_table_0 [id=0cd6d5a48cd74cff9356e87d4fadc8d3]) successfully deleted
I20240501 08:18:22.933296 27617 consensus_queue.cc:237] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:22.934978 27486 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.934970 28544 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:22.935567 29236 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.935948 29231 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:22.938683 29236 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.939154 28544 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.939198 27486 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.943547 27330 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86ccc09027574ba9b014da9428912535 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:22.944531 27345 catalog_manager.cc:5484] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 reported cstate change: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:22.947715 27728 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ccc09027574ba9b014da9428912535 with delete_type TABLET_DATA_TOMBSTONED (TS 8b39dff53c8344d3af0bf354b31998eb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35682
I20240501 08:18:22.948724 29307 tablet_replica.cc:330] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: stopping tablet replica
I20240501 08:18:22.949209 29307 raft_consensus.cc:2229] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:22.949671 29307 raft_consensus.cc:2258] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:22.950919 29307 ts_tablet_manager.cc:1875] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:22.955266 29307 ts_tablet_manager.cc:1888] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:22.955371 29307 log.cc:1196] T 86ccc09027574ba9b014da9428912535 P 8b39dff53c8344d3af0bf354b31998eb: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/86ccc09027574ba9b014da9428912535
I20240501 08:18:22.956024 27332 catalog_manager.cc:4830] TS 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383): tablet 86ccc09027574ba9b014da9428912535 (table rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]) successfully deleted
I20240501 08:18:23.067695 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:23.067910 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:23.069803 28524 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:23.071406 27728 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:23.094851 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:23.099766 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:23.183466 27750 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:23.185657 27486 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:23.185815 27617 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:23.186595 29258 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:23.186928 29253 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:23.189436 29253 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:23.190135 27486 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:23.190742 27617 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:23.195145 27342 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
W20240501 08:18:23.196250 27684 consensus_peers.cc:485] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb -> Peer 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132:34905): Couldn't send request to peer 9a400de32dd143fbbf6485d0ddc1dba7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b27438ef0d3481ca2f18ca39086af08. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:23.207108 27750 consensus_queue.cc:237] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:23.209182 27486 raft_consensus.cc:1261] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:18:23.209484 27686 consensus_peers.cc:485] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb -> Peer 40c65a4181944bf69feec459e131a224 (127.2.248.134:41033): Couldn't send request to peer 40c65a4181944bf69feec459e131a224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbc3f3548ad74a74855fd55afc53b1b2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:23.209528 27617 raft_consensus.cc:1261] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:23.209753 29253 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:23.210098 29258 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:23.212647 29183 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:23.213339 27617 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:23.213368 27486 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:23.218940 27345 catalog_manager.cc:5484] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: config changed from index -1 to 4, NON_VOTER 40c65a4181944bf69feec459e131a224 (127.2.248.134) added. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } } }
I20240501 08:18:23.229223 27486 consensus_queue.cc:237] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:23.231854 27617 raft_consensus.cc:1261] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:23.232349 27750 raft_consensus.cc:1261] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:23.232522 28311 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:23.233080 29373 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:23.236584 28311 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:23.239524 27750 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
W20240501 08:18:23.241853 27421 consensus_peers.cc:485] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 -> Peer 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132:34905): Couldn't send request to peer 9a400de32dd143fbbf6485d0ddc1dba7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08392402ccb6425aa87e1be0d435bbe6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:23.245054 27339 catalog_manager.cc:5484] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb reported cstate change: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:23.252105 27618 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } } }
I20240501 08:18:23.695000 29408 ts_tablet_manager.cc:902] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: Initiating tablet copy from peer 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543)
I20240501 08:18:23.696408 29408 tablet_copy_client.cc:308] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:44543
I20240501 08:18:23.702286 27507 tablet_copy_service.cc:140] P 64da46caf0204911aba9aba77e5ba045: Received BeginTabletCopySession request for tablet 08392402ccb6425aa87e1be0d435bbe6 from peer 9a400de32dd143fbbf6485d0ddc1dba7 ({username='slave'} at 127.2.248.132:42365)
I20240501 08:18:23.702466 27507 tablet_copy_service.cc:161] P 64da46caf0204911aba9aba77e5ba045: Beginning new tablet copy session on tablet 08392402ccb6425aa87e1be0d435bbe6 from peer 9a400de32dd143fbbf6485d0ddc1dba7 at {username='slave'} at 127.2.248.132:42365: session id = 9a400de32dd143fbbf6485d0ddc1dba7-08392402ccb6425aa87e1be0d435bbe6
I20240501 08:18:23.704361 27507 tablet_copy_source_session.cc:205] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:23.705543 29408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08392402ccb6425aa87e1be0d435bbe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:23.709798 29408 tablet_copy_client.cc:783] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:23.710068 29408 tablet_copy_client.cc:647] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:23.711822 29408 tablet_copy_client.cc:523] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:23.714501 29408 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap starting.
I20240501 08:18:23.748906 29408 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:23.749366 29408 tablet_bootstrap.cc:492] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap complete.
I20240501 08:18:23.749653 29408 ts_tablet_manager.cc:1367] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: Time spent bootstrapping tablet: real 0.035s	user 0.030s	sys 0.006s
I20240501 08:18:23.750613 29408 raft_consensus.cc:348] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:23.750823 29408 raft_consensus.cc:729] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a400de32dd143fbbf6485d0ddc1dba7, State: Initialized, Role: LEARNER
I20240501 08:18:23.751009 29408 consensus_queue.cc:260] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:23.751811 29408 ts_tablet_manager.cc:1398] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240501 08:18:23.752394 29431 ts_tablet_manager.cc:902] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: Initiating tablet copy from peer 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:23.752545 27507 tablet_copy_service.cc:342] P 64da46caf0204911aba9aba77e5ba045: Request end of tablet copy session 9a400de32dd143fbbf6485d0ddc1dba7-08392402ccb6425aa87e1be0d435bbe6 received from {username='slave'} at 127.2.248.132:42365
I20240501 08:18:23.752635 27507 tablet_copy_service.cc:433] P 64da46caf0204911aba9aba77e5ba045: ending tablet copy session 9a400de32dd143fbbf6485d0ddc1dba7-08392402ccb6425aa87e1be0d435bbe6 on tablet 08392402ccb6425aa87e1be0d435bbe6 with peer 9a400de32dd143fbbf6485d0ddc1dba7
I20240501 08:18:23.754904 29431 tablet_copy_client.cc:308] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:38383
I20240501 08:18:23.760521 27770 tablet_copy_service.cc:140] P 8b39dff53c8344d3af0bf354b31998eb: Received BeginTabletCopySession request for tablet 1b27438ef0d3481ca2f18ca39086af08 from peer 9a400de32dd143fbbf6485d0ddc1dba7 ({username='slave'} at 127.2.248.132:51189)
I20240501 08:18:23.760680 27770 tablet_copy_service.cc:161] P 8b39dff53c8344d3af0bf354b31998eb: Beginning new tablet copy session on tablet 1b27438ef0d3481ca2f18ca39086af08 from peer 9a400de32dd143fbbf6485d0ddc1dba7 at {username='slave'} at 127.2.248.132:51189: session id = 9a400de32dd143fbbf6485d0ddc1dba7-1b27438ef0d3481ca2f18ca39086af08
I20240501 08:18:23.762435 27770 tablet_copy_source_session.cc:205] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:23.763417 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b27438ef0d3481ca2f18ca39086af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:23.767195 29431 tablet_copy_client.cc:783] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:23.767410 29431 tablet_copy_client.cc:647] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:23.768823 29431 tablet_copy_client.cc:523] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:23.771027 29431 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap starting.
I20240501 08:18:23.800434 29431 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:23.800799 29431 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: Bootstrap complete.
I20240501 08:18:23.801026 29431 ts_tablet_manager.cc:1367] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: Time spent bootstrapping tablet: real 0.030s	user 0.022s	sys 0.009s
I20240501 08:18:23.801612 29431 raft_consensus.cc:348] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:23.801852 29431 raft_consensus.cc:729] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a400de32dd143fbbf6485d0ddc1dba7, State: Initialized, Role: LEARNER
I20240501 08:18:23.802052 29431 consensus_queue.cc:260] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: NON_VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: true } }
I20240501 08:18:23.802526 29431 ts_tablet_manager.cc:1398] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:18:23.803215 27770 tablet_copy_service.cc:342] P 8b39dff53c8344d3af0bf354b31998eb: Request end of tablet copy session 9a400de32dd143fbbf6485d0ddc1dba7-1b27438ef0d3481ca2f18ca39086af08 received from {username='slave'} at 127.2.248.132:51189
I20240501 08:18:23.803344 27770 tablet_copy_service.cc:433] P 8b39dff53c8344d3af0bf354b31998eb: ending tablet copy session 9a400de32dd143fbbf6485d0ddc1dba7-1b27438ef0d3481ca2f18ca39086af08 on tablet 1b27438ef0d3481ca2f18ca39086af08 with peer 9a400de32dd143fbbf6485d0ddc1dba7
I20240501 08:18:23.806697 29437 ts_tablet_manager.cc:902] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: Initiating tablet copy from peer 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:23.807713 29437 tablet_copy_client.cc:308] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:38383
I20240501 08:18:23.808346 27770 tablet_copy_service.cc:140] P 8b39dff53c8344d3af0bf354b31998eb: Received BeginTabletCopySession request for tablet dbc3f3548ad74a74855fd55afc53b1b2 from peer 40c65a4181944bf69feec459e131a224 ({username='slave'} at 127.2.248.134:39835)
I20240501 08:18:23.808497 27770 tablet_copy_service.cc:161] P 8b39dff53c8344d3af0bf354b31998eb: Beginning new tablet copy session on tablet dbc3f3548ad74a74855fd55afc53b1b2 from peer 40c65a4181944bf69feec459e131a224 at {username='slave'} at 127.2.248.134:39835: session id = 40c65a4181944bf69feec459e131a224-dbc3f3548ad74a74855fd55afc53b1b2
I20240501 08:18:23.810631 27770 tablet_copy_source_session.cc:205] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:23.811643 29437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc3f3548ad74a74855fd55afc53b1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:23.815797 29437 tablet_copy_client.cc:783] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:23.816008 29437 tablet_copy_client.cc:647] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:23.817579 29437 tablet_copy_client.cc:523] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:23.819697 29437 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: Bootstrap starting.
I20240501 08:18:23.846618 29437 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:23.847028 29437 tablet_bootstrap.cc:492] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: Bootstrap complete.
I20240501 08:18:23.847272 29437 ts_tablet_manager.cc:1367] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: Time spent bootstrapping tablet: real 0.028s	user 0.026s	sys 0.000s
I20240501 08:18:23.847901 29437 raft_consensus.cc:348] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:23.848095 29437 raft_consensus.cc:729] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40c65a4181944bf69feec459e131a224, State: Initialized, Role: LEARNER
I20240501 08:18:23.848309 29437 consensus_queue.cc:260] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: NON_VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: true } }
I20240501 08:18:23.849057 29437 ts_tablet_manager.cc:1398] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.004s
I20240501 08:18:23.849882 27770 tablet_copy_service.cc:342] P 8b39dff53c8344d3af0bf354b31998eb: Request end of tablet copy session 40c65a4181944bf69feec459e131a224-dbc3f3548ad74a74855fd55afc53b1b2 received from {username='slave'} at 127.2.248.134:39835
I20240501 08:18:23.850031 27770 tablet_copy_service.cc:433] P 8b39dff53c8344d3af0bf354b31998eb: ending tablet copy session 40c65a4181944bf69feec459e131a224-dbc3f3548ad74a74855fd55afc53b1b2 on tablet dbc3f3548ad74a74855fd55afc53b1b2 with peer 40c65a4181944bf69feec459e131a224
I20240501 08:18:24.174163 28544 raft_consensus.cc:1203] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:24.283417 28544 raft_consensus.cc:1203] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:24.340246 28804 raft_consensus.cc:1203] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:24.744896 29460 raft_consensus.cc:1050] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045: attempting to promote NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 to VOTER
I20240501 08:18:24.745579 29460 consensus_queue.cc:237] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:24.747716 28544 raft_consensus.cc:1261] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.748107 27750 raft_consensus.cc:1261] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.748301 27618 raft_consensus.cc:1261] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.748318 29460 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.748800 29510 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.749171 29460 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.751787 29460 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.752552 28544 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.753234 27750 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.757155 27342 catalog_manager.cc:5484] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 reported cstate change: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.760906 27618 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.765692 29477 raft_consensus.cc:1050] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: attempting to promote NON_VOTER 9a400de32dd143fbbf6485d0ddc1dba7 to VOTER
I20240501 08:18:24.766402 29477 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:24.768919 28544 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 LEARNER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.769084 27618 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.769477 29253 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.769896 29476 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.772225 27486 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.772784 29476 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.779295 27487 consensus_queue.cc:237] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:24.779755 29253 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.781029 27486 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.781164 28544 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.783668 28543 raft_consensus.cc:1261] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:24.784320 29460 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:24.785063 27618 raft_consensus.cc:1261] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 64da46caf0204911aba9aba77e5ba045: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:24.785872 29510 consensus_queue.cc:1035] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:24.786652 27341 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 reported cstate change: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.787302 27618 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.787465 29510 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.791877 27332 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08392402ccb6425aa87e1be0d435bbe6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:24.792445 27618 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.792965 28544 raft_consensus.cc:2932] T 08392402ccb6425aa87e1be0d435bbe6 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:24.795102 27341 catalog_manager.cc:5484] T 08392402ccb6425aa87e1be0d435bbe6 P 64da46caf0204911aba9aba77e5ba045 reported cstate change: config changed from index 5 to 6, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New cstate: current_term: 1 leader_uuid: "64da46caf0204911aba9aba77e5ba045" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:24.798753 27728 tablet_service.cc:1514] Processing DeleteTablet for tablet 08392402ccb6425aa87e1be0d435bbe6 with delete_type TABLET_DATA_TOMBSTONED (TS 8b39dff53c8344d3af0bf354b31998eb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35682
I20240501 08:18:24.799427 29523 tablet_replica.cc:330] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: stopping tablet replica
I20240501 08:18:24.799772 29523 raft_consensus.cc:2229] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:24.800156 29523 raft_consensus.cc:2258] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:24.800876 29523 ts_tablet_manager.cc:1875] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:24.805006 29523 ts_tablet_manager.cc:1888] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:24.805094 29523 log.cc:1196] T 08392402ccb6425aa87e1be0d435bbe6 P 8b39dff53c8344d3af0bf354b31998eb: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/08392402ccb6425aa87e1be0d435bbe6
I20240501 08:18:24.805723 27332 catalog_manager.cc:4830] TS 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383): tablet 08392402ccb6425aa87e1be0d435bbe6 (table rebalance_test_table_2 [id=58e7f5c394eb4568966234acac1546a9]) successfully deleted
I20240501 08:18:24.895586 27750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08"
dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:57158
I20240501 08:18:24.895853 27750 raft_consensus.cc:595] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:24.904536 29477 raft_consensus.cc:1050] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb: attempting to promote NON_VOTER 40c65a4181944bf69feec459e131a224 to VOTER
I20240501 08:18:24.905172 29477 consensus_queue.cc:237] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } }
I20240501 08:18:24.907083 27618 raft_consensus.cc:1261] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.907212 28804 raft_consensus.cc:1261] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [term 1 LEARNER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.907757 27486 raft_consensus.cc:1261] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:24.907757 29477 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.908071 29253 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.908414 29476 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:24.922544 29476 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.923357 27618 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.923777 27486 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.928243 28804 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.929050 27339 catalog_manager.cc:5484] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: config changed from index 4 to 5, 40c65a4181944bf69feec459e131a224 (127.2.248.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } attrs { replace: true } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.939991 27750 consensus_queue.cc:237] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } }
I20240501 08:18:24.941862 28804 raft_consensus.cc:1261] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:24.942292 29556 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:24.942893 27618 raft_consensus.cc:1261] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Refusing update from remote peer 8b39dff53c8344d3af0bf354b31998eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:24.944461 29477 consensus_queue.cc:1035] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:24.945063 29477 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.945603 28804 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.948455 27332 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dbc3f3548ad74a74855fd55afc53b1b2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:24.951308 27345 catalog_manager.cc:5484] T dbc3f3548ad74a74855fd55afc53b1b2 P 40c65a4181944bf69feec459e131a224 reported cstate change: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "8b39dff53c8344d3af0bf354b31998eb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.953226 27618 raft_consensus.cc:2932] T dbc3f3548ad74a74855fd55afc53b1b2 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64da46caf0204911aba9aba77e5ba045 (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } } peers { permanent_uuid: "40c65a4181944bf69feec459e131a224" member_type: VOTER last_known_addr { host: "127.2.248.134" port: 41033 } attrs { promote: false } } }
I20240501 08:18:24.954109 27750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08"
dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:57182
I20240501 08:18:24.954313 27750 raft_consensus.cc:595] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:24.955605 27466 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc3f3548ad74a74855fd55afc53b1b2 with delete_type TABLET_DATA_TOMBSTONED (TS 64da46caf0204911aba9aba77e5ba045 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42448
I20240501 08:18:24.958317 29565 tablet_replica.cc:330] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: stopping tablet replica
I20240501 08:18:24.958495 29565 raft_consensus.cc:2229] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:24.958899 29565 raft_consensus.cc:2258] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:24.959653 29565 ts_tablet_manager.cc:1875] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:24.964040 29565 ts_tablet_manager.cc:1888] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:24.964156 29565 log.cc:1196] T dbc3f3548ad74a74855fd55afc53b1b2 P 64da46caf0204911aba9aba77e5ba045: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbc3f3548ad74a74855fd55afc53b1b2
I20240501 08:18:24.964831 27332 catalog_manager.cc:4830] TS 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543): tablet dbc3f3548ad74a74855fd55afc53b1b2 (table rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]) successfully deleted
I20240501 08:18:25.180068 27750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08"
dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:57200
I20240501 08:18:25.180277 27750 raft_consensus.cc:595] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:25.195600 27750 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08"
dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:57208
I20240501 08:18:25.195747 27750 raft_consensus.cc:595] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:25.208170 29556 raft_consensus.cc:979] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: : Instructing follower 9a400de32dd143fbbf6485d0ddc1dba7 to start an election
I20240501 08:18:25.208328 29477 raft_consensus.cc:1067] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Signalling peer 9a400de32dd143fbbf6485d0ddc1dba7 to start an election
I20240501 08:18:25.209064 28544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08"
dest_uuid: "9a400de32dd143fbbf6485d0ddc1dba7"
 from {username='slave'} at 127.2.248.131:37409
I20240501 08:18:25.209317 28544 raft_consensus.cc:482] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:25.209510 28544 raft_consensus.cc:3037] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:25.211416 28544 raft_consensus.cc:504] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:25.212960 28544 leader_election.cc:288] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [CANDIDATE]: Term 2 election: Requested vote from peers 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543), dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823), 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383)
I20240501 08:18:25.213287 27486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08" candidate_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "64da46caf0204911aba9aba77e5ba045"
I20240501 08:18:25.213498 27486 raft_consensus.cc:3037] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:25.213421 27618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08" candidate_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b"
I20240501 08:18:25.213637 27618 raft_consensus.cc:3037] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:25.213599 27750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b27438ef0d3481ca2f18ca39086af08" candidate_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8b39dff53c8344d3af0bf354b31998eb"
I20240501 08:18:25.213785 27750 raft_consensus.cc:3032] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:25.213861 27750 raft_consensus.cc:729] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Running, Role: LEADER
I20240501 08:18:25.214159 27750 consensus_queue.cc:260] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:25.214578 27750 raft_consensus.cc:3037] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:25.215206 27486 raft_consensus.cc:2454] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a400de32dd143fbbf6485d0ddc1dba7 in term 2.
I20240501 08:18:25.216305 27618 raft_consensus.cc:2454] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a400de32dd143fbbf6485d0ddc1dba7 in term 2.
I20240501 08:18:25.216750 28477 leader_election.cc:302] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 64da46caf0204911aba9aba77e5ba045, 9a400de32dd143fbbf6485d0ddc1dba7, dc0e586a35fa4370bcd23da85a3a4b0b; no voters: 
I20240501 08:18:25.217026 27750 raft_consensus.cc:2454] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a400de32dd143fbbf6485d0ddc1dba7 in term 2.
I20240501 08:18:25.217056 29430 raft_consensus.cc:2781] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:25.217507 29430 raft_consensus.cc:686] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 2 LEADER]: Becoming Leader. State: Replica: 9a400de32dd143fbbf6485d0ddc1dba7, State: Running, Role: LEADER
I20240501 08:18:25.217962 29430 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:25.221185 27345 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 reported cstate change: term changed from 1 to 2, leader changed from 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) to 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132). New cstate: current_term: 2 leader_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:25.619419 27750 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:25.620146 29430 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:25.718421 27618 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:25.718991 29430 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:25.723672 27486 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:25.724241 29603 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:26.253649 28544 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } }
I20240501 08:18:26.255285 27486 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:18:26.255342 27618 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:18:26.255916 29625 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:26.256304 29624 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:26.258642 29603 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:26.259249 27618 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:26.260352 27486 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } }
I20240501 08:18:26.262575 27331 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b27438ef0d3481ca2f18ca39086af08 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:26.264303 27339 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 reported cstate change: config changed from index 5 to 7, VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) evicted. New cstate: current_term: 2 leader_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:26.268875 27728 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b27438ef0d3481ca2f18ca39086af08 with delete_type TABLET_DATA_TOMBSTONED (TS 8b39dff53c8344d3af0bf354b31998eb not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35682
I20240501 08:18:26.269538 29631 tablet_replica.cc:330] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: stopping tablet replica
I20240501 08:18:26.269742 29631 raft_consensus.cc:2229] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:26.269977 29631 raft_consensus.cc:2258] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:26.270768 29631 ts_tablet_manager.cc:1875] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:26.274932 29631 ts_tablet_manager.cc:1888] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:18:26.275054 29631 log.cc:1196] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/1b27438ef0d3481ca2f18ca39086af08
I20240501 08:18:26.275820 27332 catalog_manager.cc:4830] TS 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383): tablet 1b27438ef0d3481ca2f18ca39086af08 (table rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]) successfully deleted
I20240501 08:18:26.544063 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.558729 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:26.571913 28524 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:26.574728 27728 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:26.603358 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.610702 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:26.754951 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.765772 28524 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:26.776101 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:26.777227 27728 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:26.796444 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.823354 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:26.886648 27618 consensus_queue.cc:237] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:26.889868 28544 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:18:26.890501 29724 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:26.891677 28543 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: true } }
I20240501 08:18:26.894910 27486 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:18:26.895921 29724 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:26.896035 27618 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:18:26.898083 29711 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:18:26.899114 28479 consensus_peers.cc:485] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 -> Peer 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131:38383): Couldn't send request to peer 8b39dff53c8344d3af0bf354b31998eb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:26.899976 27486 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 08:18:26.900079 27555 consensus_peers.cc:485] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b -> Peer b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133:42453): Couldn't send request to peer b755b190bbfc4bc0a08c28c7263c94cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86ccc09027574ba9b014da9428912535. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:26.900699 29712 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:26.902393 29712 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: true } } }
I20240501 08:18:26.902881 27618 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: true } } }
I20240501 08:18:26.906133 27486 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: true } } }
I20240501 08:18:26.909977 27341 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: config changed from index 7 to 8, NON_VOTER 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) added. New cstate: current_term: 2 leader_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: true } } }
I20240501 08:18:26.924093 29724 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:26.929234 27487 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:26.934461 28544 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } } }
I20240501 08:18:26.943222 27345 catalog_manager.cc:5484] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: config changed from index 6 to 7, NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:27.311537 29755 ts_tablet_manager.cc:902] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Initiating tablet copy from peer 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132:34905)
I20240501 08:18:27.313186 29755 tablet_copy_client.cc:308] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.132:34905
I20240501 08:18:27.314071 28564 tablet_copy_service.cc:140] P 9a400de32dd143fbbf6485d0ddc1dba7: Received BeginTabletCopySession request for tablet 1b27438ef0d3481ca2f18ca39086af08 from peer 8b39dff53c8344d3af0bf354b31998eb ({username='slave'} at 127.2.248.131:37409)
I20240501 08:18:27.314342 28564 tablet_copy_service.cc:161] P 9a400de32dd143fbbf6485d0ddc1dba7: Beginning new tablet copy session on tablet 1b27438ef0d3481ca2f18ca39086af08 from peer 8b39dff53c8344d3af0bf354b31998eb at {username='slave'} at 127.2.248.131:37409: session id = 8b39dff53c8344d3af0bf354b31998eb-1b27438ef0d3481ca2f18ca39086af08
I20240501 08:18:27.317095 28564 tablet_copy_source_session.cc:205] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:27.318791 29755 ts_tablet_manager.cc:1875] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:18:27.319691 29760 ts_tablet_manager.cc:902] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: Initiating tablet copy from peer dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823)
I20240501 08:18:27.321343 29760 tablet_copy_client.cc:308] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46823
I20240501 08:18:27.324227 29755 ts_tablet_manager.cc:1888] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 08:18:27.324544 29755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b27438ef0d3481ca2f18ca39086af08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:27.327016 29755 tablet_copy_client.cc:783] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:27.327322 29755 tablet_copy_client.cc:647] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:27.328140 27639 tablet_copy_service.cc:140] P dc0e586a35fa4370bcd23da85a3a4b0b: Received BeginTabletCopySession request for tablet 86ccc09027574ba9b014da9428912535 from peer b755b190bbfc4bc0a08c28c7263c94cd ({username='slave'} at 127.2.248.133:50979)
I20240501 08:18:27.328302 27639 tablet_copy_service.cc:161] P dc0e586a35fa4370bcd23da85a3a4b0b: Beginning new tablet copy session on tablet 86ccc09027574ba9b014da9428912535 from peer b755b190bbfc4bc0a08c28c7263c94cd at {username='slave'} at 127.2.248.133:50979: session id = b755b190bbfc4bc0a08c28c7263c94cd-86ccc09027574ba9b014da9428912535
I20240501 08:18:27.329370 29755 tablet_copy_client.cc:523] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:27.330909 27639 tablet_copy_source_session.cc:205] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:27.332154 29760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ccc09027574ba9b014da9428912535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:27.332790 29755 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap starting.
I20240501 08:18:27.336427 29760 tablet_copy_client.cc:783] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:27.336697 29760 tablet_copy_client.cc:647] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:27.338318 29760 tablet_copy_client.cc:523] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:27.340695 29760 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap starting.
I20240501 08:18:27.371922 29755 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:27.372643 29755 tablet_bootstrap.cc:492] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Bootstrap complete.
I20240501 08:18:27.373091 29755 ts_tablet_manager.cc:1367] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Time spent bootstrapping tablet: real 0.040s	user 0.036s	sys 0.004s
I20240501 08:18:27.374259 29755 raft_consensus.cc:348] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: true } }
I20240501 08:18:27.374740 29755 raft_consensus.cc:729] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8b39dff53c8344d3af0bf354b31998eb, State: Initialized, Role: LEARNER
I20240501 08:18:27.375092 29755 consensus_queue.cc:260] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: true } }
I20240501 08:18:27.376854 29755 ts_tablet_manager.cc:1398] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240501 08:18:27.377647 28564 tablet_copy_service.cc:342] P 9a400de32dd143fbbf6485d0ddc1dba7: Request end of tablet copy session 8b39dff53c8344d3af0bf354b31998eb-1b27438ef0d3481ca2f18ca39086af08 received from {username='slave'} at 127.2.248.131:37409
I20240501 08:18:27.377831 28564 tablet_copy_service.cc:433] P 9a400de32dd143fbbf6485d0ddc1dba7: ending tablet copy session 8b39dff53c8344d3af0bf354b31998eb-1b27438ef0d3481ca2f18ca39086af08 on tablet 1b27438ef0d3481ca2f18ca39086af08 with peer 8b39dff53c8344d3af0bf354b31998eb
I20240501 08:18:27.388381 29760 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:27.388793 29760 tablet_bootstrap.cc:492] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: Bootstrap complete.
I20240501 08:18:27.389041 29760 ts_tablet_manager.cc:1367] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: Time spent bootstrapping tablet: real 0.048s	user 0.045s	sys 0.001s
I20240501 08:18:27.389927 29760 raft_consensus.cc:348] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:27.390218 29760 raft_consensus.cc:729] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b755b190bbfc4bc0a08c28c7263c94cd, State: Initialized, Role: LEARNER
I20240501 08:18:27.390460 29760 consensus_queue.cc:260] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: true } }
I20240501 08:18:27.392601 29760 ts_tablet_manager.cc:1398] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:27.393251 27639 tablet_copy_service.cc:342] P dc0e586a35fa4370bcd23da85a3a4b0b: Request end of tablet copy session b755b190bbfc4bc0a08c28c7263c94cd-86ccc09027574ba9b014da9428912535 received from {username='slave'} at 127.2.248.133:50979
I20240501 08:18:27.393362 27639 tablet_copy_service.cc:433] P dc0e586a35fa4370bcd23da85a3a4b0b: ending tablet copy session b755b190bbfc4bc0a08c28c7263c94cd-86ccc09027574ba9b014da9428912535 on tablet 86ccc09027574ba9b014da9428912535 with peer b755b190bbfc4bc0a08c28c7263c94cd
I20240501 08:18:27.758244 27750 raft_consensus.cc:1203] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240501 08:18:27.896049 28674 raft_consensus.cc:1203] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 08:18:28.265159 29823 raft_consensus.cc:1050] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7: attempting to promote NON_VOTER 8b39dff53c8344d3af0bf354b31998eb to VOTER
I20240501 08:18:28.265872 29823 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } }
I20240501 08:18:28.267912 27618 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:28.267928 27750 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 LEARNER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:28.268450 27487 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:28.268440 29825 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:28.268755 29789 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:28.269096 29823 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:28.271796 29823 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.272447 27618 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.272598 27487 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.276146 27750 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.277508 27341 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 reported cstate change: config changed from index 8 to 9, 8b39dff53c8344d3af0bf354b31998eb (127.2.248.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } attrs { replace: true } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.285267 28544 consensus_queue.cc:237] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } }
I20240501 08:18:28.286957 27750 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:18:28.287266 27487 raft_consensus.cc:1261] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Refusing update from remote peer 9a400de32dd143fbbf6485d0ddc1dba7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:18:28.287729 29823 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:18:28.288028 29825 consensus_queue.cc:1035] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:18:28.290019 29825 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.291031 29801 raft_consensus.cc:1050] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b: attempting to promote NON_VOTER b755b190bbfc4bc0a08c28c7263c94cd to VOTER
I20240501 08:18:28.291623 29801 consensus_queue.cc:237] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } }
I20240501 08:18:28.293144 27331 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b27438ef0d3481ca2f18ca39086af08 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:28.293992 27750 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 8b39dff53c8344d3af0bf354b31998eb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.294137 27487 raft_consensus.cc:2932] T 1b27438ef0d3481ca2f18ca39086af08 P 64da46caf0204911aba9aba77e5ba045 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } } }
I20240501 08:18:28.294505 27342 catalog_manager.cc:5484] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 reported cstate change: config changed from index 9 to 10, VOTER dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130) evicted. New cstate: current_term: 2 leader_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b39dff53c8344d3af0bf354b31998eb" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 38383 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:28.298393 28544 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:18:28.298803 28674 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 LEARNER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:18:28.299695 29835 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:28.299952 27598 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b27438ef0d3481ca2f18ca39086af08 with delete_type TABLET_DATA_TOMBSTONED (TS dc0e586a35fa4370bcd23da85a3a4b0b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48666
I20240501 08:18:28.300644 29836 tablet_replica.cc:330] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: stopping tablet replica
I20240501 08:18:28.300832 29836 raft_consensus.cc:2229] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:28.300870 27487 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:18:28.301050 29836 raft_consensus.cc:2258] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:28.301540 29800 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:28.301885 29836 ts_tablet_manager.cc:1875] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:28.304054 29724 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:28.306661 29836 ts_tablet_manager.cc:1888] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 08:18:28.306777 29836 log.cc:1196] T 1b27438ef0d3481ca2f18ca39086af08 P dc0e586a35fa4370bcd23da85a3a4b0b: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b27438ef0d3481ca2f18ca39086af08
I20240501 08:18:28.307425 27330 catalog_manager.cc:4830] TS dc0e586a35fa4370bcd23da85a3a4b0b (127.2.248.130:46823): tablet 1b27438ef0d3481ca2f18ca39086af08 (table rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]) successfully deleted
I20240501 08:18:28.309378 29801 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.311252 28544 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.313163 28674 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.313457 27487 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.317082 27339 catalog_manager.cc:5484] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b reported cstate change: config changed from index 7 to 8, b755b190bbfc4bc0a08c28c7263c94cd (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a400de32dd143fbbf6485d0ddc1dba7" member_type: VOTER last_known_addr { host: "127.2.248.132" port: 34905 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:28.322973 27618 consensus_queue.cc:237] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } }
I20240501 08:18:28.325062 28675 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:18:28.325074 27487 raft_consensus.cc:1261] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Refusing update from remote peer dc0e586a35fa4370bcd23da85a3a4b0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:18:28.325585 29801 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:28.325857 29835 consensus_queue.cc:1035] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:28.328346 29835 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P dc0e586a35fa4370bcd23da85a3a4b0b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.328637 28674 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.331162 27330 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86ccc09027574ba9b014da9428912535 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:28.332427 27487 raft_consensus.cc:2932] T 86ccc09027574ba9b014da9428912535 P 64da46caf0204911aba9aba77e5ba045 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.332738 27339 catalog_manager.cc:5484] T 86ccc09027574ba9b014da9428912535 P b755b190bbfc4bc0a08c28c7263c94cd reported cstate change: config changed from index 8 to 9, VOTER 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132) evicted. New cstate: current_term: 1 leader_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "64da46caf0204911aba9aba77e5ba045" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 44543 } } peers { permanent_uuid: "dc0e586a35fa4370bcd23da85a3a4b0b" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46823 } } peers { permanent_uuid: "b755b190bbfc4bc0a08c28c7263c94cd" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 42453 } attrs { promote: false } } }
I20240501 08:18:28.336444 28524 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ccc09027574ba9b014da9428912535 with delete_type TABLET_DATA_TOMBSTONED (TS 9a400de32dd143fbbf6485d0ddc1dba7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55042
I20240501 08:18:28.337507 29840 tablet_replica.cc:330] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: stopping tablet replica
I20240501 08:18:28.338169 29840 raft_consensus.cc:2229] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:28.338590 29840 raft_consensus.cc:2258] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:28.339735 29840 ts_tablet_manager.cc:1875] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:28.344600 29840 ts_tablet_manager.cc:1888] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:18:28.344714 29840 log.cc:1196] T 86ccc09027574ba9b014da9428912535 P 9a400de32dd143fbbf6485d0ddc1dba7: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/wals/86ccc09027574ba9b014da9428912535
I20240501 08:18:28.345445 27331 catalog_manager.cc:4830] TS 9a400de32dd143fbbf6485d0ddc1dba7 (127.2.248.132:34905): tablet 86ccc09027574ba9b014da9428912535 (table rebalance_test_table_1 [id=7cd9975999934fd4a70da7dcef94ac43]) successfully deleted
I20240501 08:18:28.579209 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:28.579813 28524 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:28.582401 27728 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:28.584411 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:28.584486 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:28.586153 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:28.751399 28785 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:28.768450 27466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:28.774678 27728 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:28.787706 28524 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:28.799652 28655 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:28.800240 27598 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:29.009295  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 27405
I20240501 08:18:29.020087 27526 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:29.148661  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 27405
I20240501 08:18:29.173890  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 27536
I20240501 08:18:29.179710 27658 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:18:29.283099 28479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44543: connect: Connection refused (error 111)
W20240501 08:18:29.284925 28479 consensus_peers.cc:485] T 1b27438ef0d3481ca2f18ca39086af08 P 9a400de32dd143fbbf6485d0ddc1dba7 -> Peer 64da46caf0204911aba9aba77e5ba045 (127.2.248.129:44543): Couldn't send request to peer 64da46caf0204911aba9aba77e5ba045. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:44543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:29.313338  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 27536
W20240501 08:18:29.338580 28608 connection.cc:535] server connection from 127.2.248.130:38743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:18:29.339020  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 27668
I20240501 08:18:29.344907 27789 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:29.483222  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 27668
I20240501 08:18:29.504796  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 28462
I20240501 08:18:29.510430 28583 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:29.633899  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 28462
I20240501 08:18:29.654660  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 28593
I20240501 08:18:29.660550 28714 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:29.767066  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 28593
I20240501 08:18:29.787003  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 28723
I20240501 08:18:29.793187 28844 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:29.898346  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 28723
I20240501 08:18:29.918262  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 27315
I20240501 08:18:29.919397 27376 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:30.050066  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 27315
2024-05-01T08:18:30Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (23032 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240501 08:18:30.105273  3042 test_util.cc:268] Using random seed: 565542053
I20240501 08:18:30.106997  3042 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:18:30.107056  3042 ts_itest-base.cc:113] --------------
I20240501 08:18:30.107098  3042 ts_itest-base.cc:114] 5 tablet servers
I20240501 08:18:30.107136  3042 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:18:30.107180  3042 ts_itest-base.cc:116] --------------
2024-05-01T08:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:18:30Z Disabled control of system clock
I20240501 08:18:30.151252  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:35115
--webserver_interface=127.2.248.190
--webserver_port=0
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.248.190:35115
--location_mapping_cmd=/tmp/dist-test-task3vHYYh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 08:18:30.470221 29939 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 08:18:30.470566 29939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:30.470666 29939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:30.470768 29939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:30.478011 29939 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:30.478161 29939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:18:30.478266 29939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:30.478329 29939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:18:30.478433 29939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:18:30.487293 29939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task3vHYYh/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.2.248.190:35115
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.248.190:35115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.248.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:30.488608 29939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:30.490296 29939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:30.499898 29945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:30.500430 29944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:30.500721 29947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:30.501536 29939 server_base.cc:1034] running on GCE node
I20240501 08:18:30.502218 29939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:30.503463 29939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:30.504671 29939 hybrid_clock.cc:648] HybridClock initialized: now 1714551510504601 us; error 43 us; skew 500 ppm
I20240501 08:18:30.504940 29939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:30.511485 29939 webserver.cc:460] Webserver started at http://127.2.248.190:42995/ using document root <none> and password file <none>
I20240501 08:18:30.512161 29939 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:30.512284 29939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:30.512552 29939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:30.514896 29939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7cfb21beda6445deb000c6f6bac12032"
format_stamp: "Formatted at 2024-05-01 08:18:30 on dist-test-slave-0pdg"
I20240501 08:18:30.515597 29939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7cfb21beda6445deb000c6f6bac12032"
format_stamp: "Formatted at 2024-05-01 08:18:30 on dist-test-slave-0pdg"
I20240501 08:18:30.520279 29939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.007s
I20240501 08:18:30.523331 29953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:30.524518 29939 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:30.524662 29939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7cfb21beda6445deb000c6f6bac12032"
format_stamp: "Formatted at 2024-05-01 08:18:30 on dist-test-slave-0pdg"
I20240501 08:18:30.524816 29939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:30.551370 29939 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:30.552068 29939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:30.552352 29939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:30.569628 29939 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.190:35115
I20240501 08:18:30.569658 30004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.190:35115 every 8 connection(s)
I20240501 08:18:30.570894 29939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:18:30.574463  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 29939
I20240501 08:18:30.574610 30005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:30.574723  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:18:30.586177 30005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032: Bootstrap starting.
I20240501 08:18:30.589462 30005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:30.590436 30005 log.cc:824] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:30.592736 30005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032: No bootstrap required, opened a new log
I20240501 08:18:30.598762 30005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb21beda6445deb000c6f6bac12032" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 35115 } }
I20240501 08:18:30.599066 30005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:30.599146 30005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfb21beda6445deb000c6f6bac12032, State: Initialized, Role: FOLLOWER
I20240501 08:18:30.599685 30005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb21beda6445deb000c6f6bac12032" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 35115 } }
I20240501 08:18:30.599905 30005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:18:30.600049 30005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:18:30.600224 30005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:30.602291 30005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb21beda6445deb000c6f6bac12032" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 35115 } }
I20240501 08:18:30.602773 30005 leader_election.cc:302] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cfb21beda6445deb000c6f6bac12032; no voters: 
I20240501 08:18:30.603318 30005 leader_election.cc:288] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:18:30.603447 30010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:30.604713 30010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [term 1 LEADER]: Becoming Leader. State: Replica: 7cfb21beda6445deb000c6f6bac12032, State: Running, Role: LEADER
I20240501 08:18:30.605278 30010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb21beda6445deb000c6f6bac12032" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 35115 } }
I20240501 08:18:30.605455 30005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:18:30.607985 30011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cfb21beda6445deb000c6f6bac12032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb21beda6445deb000c6f6bac12032" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 35115 } } }
I20240501 08:18:30.607986 30012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cfb21beda6445deb000c6f6bac12032. Latest consensus state: current_term: 1 leader_uuid: "7cfb21beda6445deb000c6f6bac12032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfb21beda6445deb000c6f6bac12032" member_type: VOTER last_known_addr { host: "127.2.248.190" port: 35115 } } }
I20240501 08:18:30.608366 30012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [sys.catalog]: This master's current role is: LEADER
I20240501 08:18:30.608367 30011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032 [sys.catalog]: This master's current role is: LEADER
I20240501 08:18:30.610016 30016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:18:30.614612 30016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:18:30.623127 30016 catalog_manager.cc:1347] Generated new cluster ID: 8893c673d6b342ec992ac4b69a623b5f
I20240501 08:18:30.623221 30016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:18:30.636056 30016 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:18:30.637082 30016 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:18:30.646746 30016 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7cfb21beda6445deb000c6f6bac12032: Generated new TSK 0
I20240501 08:18:30.647419 30016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:18:30.655239  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.129:0
--local_ip_for_outbound_sockets=127.2.248.129
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:18:31.029687 30029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:31.029994 30029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:31.030130 30029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:31.037178 30029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:31.037285 30029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:31.037431 30029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.129
I20240501 08:18:31.045810 30029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.248.129
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.129
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:31.047061 30029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:31.048687 30029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:31.060305 30037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:31.060477 30035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:31.060627 30034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:31.061254 30029 server_base.cc:1034] running on GCE node
I20240501 08:18:31.061820 30029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:31.062696 30029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:31.063910 30029 hybrid_clock.cc:648] HybridClock initialized: now 1714551511063870 us; error 38 us; skew 500 ppm
I20240501 08:18:31.064258 30029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:31.066279 30029 webserver.cc:460] Webserver started at http://127.2.248.129:35887/ using document root <none> and password file <none>
I20240501 08:18:31.066947 30029 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:31.067056 30029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:31.067333 30029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:31.069694 30029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "23d1b1dde90249cf82ab29f1b44795de"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.070459 30029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "23d1b1dde90249cf82ab29f1b44795de"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.075150 30029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20240501 08:18:31.078239 30043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.079342 30029 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:31.079488 30029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "23d1b1dde90249cf82ab29f1b44795de"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.079660 30029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:31.092280 30029 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:31.092929 30029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:31.093155 30029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:31.093986 30029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:31.095382 30029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:31.095479 30029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.095584 30029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:31.095659 30029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.124759 30029 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.129:34997
I20240501 08:18:31.124809 30155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.129:34997 every 8 connection(s)
I20240501 08:18:31.125941 30029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:18:31.136286  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 30029
I20240501 08:18:31.136503  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:18:31.137984 30156 heartbeater.cc:347] Connected to a master server at 127.2.248.190:35115
I20240501 08:18:31.138278 30156 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:31.139158 30156 heartbeater.cc:511] Master 127.2.248.190:35115 requested a full tablet report, sending...
I20240501 08:18:31.139987  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.130:0
--local_ip_for_outbound_sockets=127.2.248.130
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:31.187891 29970 ts_manager.cc:194] Registered new tserver with Master: 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997)
I20240501 08:18:31.189747 29970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.129:60053
W20240501 08:18:31.456087 30161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:31.456363 30161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:31.456482 30161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:31.462855 30161 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:31.462941 30161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:31.463037 30161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.130
I20240501 08:18:31.470830 30161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.248.130
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.130
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:31.471937 30161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:31.473485 30161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:31.484709 30169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:31.484745 30166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:31.485172 30167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:31.485661 30161 server_base.cc:1034] running on GCE node
I20240501 08:18:31.486146 30161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:31.486927 30161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:31.488106 30161 hybrid_clock.cc:648] HybridClock initialized: now 1714551511488066 us; error 36 us; skew 500 ppm
I20240501 08:18:31.488390 30161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:31.490401 30161 webserver.cc:460] Webserver started at http://127.2.248.130:34271/ using document root <none> and password file <none>
I20240501 08:18:31.491071 30161 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:31.491180 30161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:31.491461 30161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:31.493808 30161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.494495 30161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.499130 30161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240501 08:18:31.502394 30175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.503855 30161 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:31.504033 30161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.504263 30161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:31.533618 30161 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:31.534437 30161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:31.534719 30161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:31.535630 30161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:31.537099 30161 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:31.537207 30161 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.537328 30161 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:31.537406 30161 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.569484 30161 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.130:46021
I20240501 08:18:31.569521 30287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.130:46021 every 8 connection(s)
I20240501 08:18:31.571213 30161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:18:31.579593  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 30161
I20240501 08:18:31.579797  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:18:31.583601  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.131:0
--local_ip_for_outbound_sockets=127.2.248.131
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:31.585238 30288 heartbeater.cc:347] Connected to a master server at 127.2.248.190:35115
I20240501 08:18:31.585530 30288 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:31.586447 30288 heartbeater.cc:511] Master 127.2.248.190:35115 requested a full tablet report, sending...
I20240501 08:18:31.632223 29970 ts_manager.cc:194] Registered new tserver with Master: 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:31.633450 29970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.130:36535
W20240501 08:18:31.899183 30291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:31.899427 30291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:31.899504 30291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:31.906733 30291 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:31.906831 30291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:31.907008 30291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.131
I20240501 08:18:31.915477 30291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.248.131
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.131
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:31.916756 30291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:31.918332 30291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:31.929385 30297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:31.929858 30298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:31.930028 30300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:31.930558 30291 server_base.cc:1034] running on GCE node
I20240501 08:18:31.931135 30291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:31.931989 30291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:31.933216 30291 hybrid_clock.cc:648] HybridClock initialized: now 1714551511933151 us; error 42 us; skew 500 ppm
I20240501 08:18:31.933550 30291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:31.935627 30291 webserver.cc:460] Webserver started at http://127.2.248.131:46019/ using document root <none> and password file <none>
I20240501 08:18:31.936329 30291 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:31.936436 30291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:31.936750 30291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:31.939061 30291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7e7b5a0a22114297942fe102e2cbd890"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.939743 30291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7e7b5a0a22114297942fe102e2cbd890"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.944300 30291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.003s
I20240501 08:18:31.947252 30306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.948401 30291 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20240501 08:18:31.948536 30291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7e7b5a0a22114297942fe102e2cbd890"
format_stamp: "Formatted at 2024-05-01 08:18:31 on dist-test-slave-0pdg"
I20240501 08:18:31.948711 30291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:31.977239 30291 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:31.977958 30291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:31.978173 30291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:31.979024 30291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:31.980388 30291 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:31.980471 30291 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:31.980551 30291 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:31.980604 30291 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:32.010605 30291 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.131:34593
I20240501 08:18:32.010692 30418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.131:34593 every 8 connection(s)
I20240501 08:18:32.011806 30291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:18:32.012917  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 30291
I20240501 08:18:32.013135  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:18:32.016700  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:0
--local_ip_for_outbound_sockets=127.2.248.132
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:32.025497 30419 heartbeater.cc:347] Connected to a master server at 127.2.248.190:35115
I20240501 08:18:32.025758 30419 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:32.026613 30419 heartbeater.cc:511] Master 127.2.248.190:35115 requested a full tablet report, sending...
I20240501 08:18:32.085186 29970 ts_manager.cc:194] Registered new tserver with Master: 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:32.086237 29970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.131:50725
I20240501 08:18:32.193221 30156 heartbeater.cc:503] Master 127.2.248.190:35115 was elected leader, sending a full tablet report...
W20240501 08:18:32.359283 30422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:32.359539 30422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:32.359622 30422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:32.366628 30422 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:32.366744 30422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:32.366864 30422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:18:32.375108 30422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:32.376246 30422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:32.377861 30422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:32.389197 30431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:32.389688 30428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:32.390249 30422 server_base.cc:1034] running on GCE node
W20240501 08:18:32.390617 30429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:32.391202 30422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:32.392081 30422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:32.393307 30422 hybrid_clock.cc:648] HybridClock initialized: now 1714551512393253 us; error 50 us; skew 500 ppm
I20240501 08:18:32.393605 30422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:32.395731 30422 webserver.cc:460] Webserver started at http://127.2.248.132:33097/ using document root <none> and password file <none>
I20240501 08:18:32.396489 30422 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:32.396600 30422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:32.396891 30422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:32.399319 30422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0492adfedf2f446dbfc904261c50475c"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:32.400074 30422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0492adfedf2f446dbfc904261c50475c"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:32.404938 30422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:18:32.407992 30437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:32.409288 30422 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:32.409442 30422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0492adfedf2f446dbfc904261c50475c"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:32.409655 30422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:32.427191 30422 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:32.427917 30422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:32.428174 30422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:32.428982 30422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:32.430359 30422 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:32.430440 30422 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:32.430516 30422 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:32.430568 30422 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:32.460873 30422 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:46099
I20240501 08:18:32.460955 30549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:46099 every 8 connection(s)
I20240501 08:18:32.462069 30422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:18:32.468273  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 30422
I20240501 08:18:32.468458  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:18:32.471985  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:0
--local_ip_for_outbound_sockets=127.2.248.133
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:32.475376 30550 heartbeater.cc:347] Connected to a master server at 127.2.248.190:35115
I20240501 08:18:32.475687 30550 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:32.476693 30550 heartbeater.cc:511] Master 127.2.248.190:35115 requested a full tablet report, sending...
I20240501 08:18:32.537303 29970 ts_manager.cc:194] Registered new tserver with Master: 0492adfedf2f446dbfc904261c50475c (127.2.248.132:46099)
I20240501 08:18:32.538233 29970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:48775
I20240501 08:18:32.636168 30288 heartbeater.cc:503] Master 127.2.248.190:35115 was elected leader, sending a full tablet report...
W20240501 08:18:32.813637 30553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:32.813915 30553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:32.814051 30553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:32.820958 30553 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:32.821084 30553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:32.821240 30553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:18:32.829502 30553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=0
--tserver_master_addrs=127.2.248.190:35115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:32.830747 30553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:32.832429 30553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:32.843446 30559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:32.843891 30562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:32.844355 30560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:32.844741 30553 server_base.cc:1034] running on GCE node
I20240501 08:18:32.845250 30553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:32.846055 30553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:32.847239 30553 hybrid_clock.cc:648] HybridClock initialized: now 1714551512847193 us; error 39 us; skew 500 ppm
I20240501 08:18:32.847499 30553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:32.849439 30553 webserver.cc:460] Webserver started at http://127.2.248.133:33629/ using document root <none> and password file <none>
I20240501 08:18:32.850102 30553 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:32.850212 30553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:32.850481 30553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:18:32.852878 30553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "309c676e56d04218a47e557e6a834030"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:32.853634 30553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "309c676e56d04218a47e557e6a834030"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:32.858215 30553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240501 08:18:32.861398 30568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:32.862535 30553 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I20240501 08:18:32.862679 30553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "309c676e56d04218a47e557e6a834030"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:32.862862 30553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:32.887216 30553 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:32.887972 30553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:32.888232 30553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:32.889143 30553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:32.890501 30553 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:32.890586 30553 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:32.890691 30553 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:32.890769 30553 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:32.920912 30553 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:35633
I20240501 08:18:32.920960 30680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:35633 every 8 connection(s)
I20240501 08:18:32.922133 30553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:18:32.924403  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 30553
I20240501 08:18:32.924633  3042 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:18:32.934188 30681 heartbeater.cc:347] Connected to a master server at 127.2.248.190:35115
I20240501 08:18:32.934464 30681 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:32.935197 30681 heartbeater.cc:511] Master 127.2.248.190:35115 requested a full tablet report, sending...
I20240501 08:18:32.978806 29970 ts_manager.cc:194] Registered new tserver with Master: 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633)
I20240501 08:18:32.979974 29970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:42485
I20240501 08:18:32.980866  3042 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:18:32.996851  3042 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:18:32.996964  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 30553
I20240501 08:18:33.004185 30676 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:33.083460  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 30553
I20240501 08:18:33.088852 30419 heartbeater.cc:503] Master 127.2.248.190:35115 was elected leader, sending a full tablet report...
I20240501 08:18:33.099453  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 30422
I20240501 08:18:33.107352 30545 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:33.186268  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 30422
I20240501 08:18:36.954463 29970 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33194:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 08:18:36.960996 29970 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:18:36.990363 30353 tablet_service.cc:1467] Processing CreateTablet for tablet 844293e1c98e41b1bcb772284d7d7075 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.990490 30221 tablet_service.cc:1467] Processing CreateTablet for tablet 2a85f5b4f62343a4812d8c1f75c8694a (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.990360 30090 tablet_service.cc:1467] Processing CreateTablet for tablet 844293e1c98e41b1bcb772284d7d7075 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.990859 30222 tablet_service.cc:1467] Processing CreateTablet for tablet 844293e1c98e41b1bcb772284d7d7075 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.991721 30223 tablet_service.cc:1467] Processing CreateTablet for tablet 80b5f6bc3ac842ff931e03ae3094df55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.991891 30090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844293e1c98e41b1bcb772284d7d7075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:36.991895 30222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844293e1c98e41b1bcb772284d7d7075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:36.991905 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844293e1c98e41b1bcb772284d7d7075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:36.992261 30354 tablet_service.cc:1467] Processing CreateTablet for tablet 80b5f6bc3ac842ff931e03ae3094df55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.992542 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 2a85f5b4f62343a4812d8c1f75c8694a (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.992969 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a85f5b4f62343a4812d8c1f75c8694a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:36.994488 30354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b5f6bc3ac842ff931e03ae3094df55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:36.996512 30223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b5f6bc3ac842ff931e03ae3094df55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:36.996619 30089 tablet_service.cc:1467] Processing CreateTablet for tablet 2a85f5b4f62343a4812d8c1f75c8694a (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.997097 30089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a85f5b4f62343a4812d8c1f75c8694a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:36.998023 30091 tablet_service.cc:1467] Processing CreateTablet for tablet 80b5f6bc3ac842ff931e03ae3094df55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:36.998411 30091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b5f6bc3ac842ff931e03ae3094df55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:37.000054 30221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a85f5b4f62343a4812d8c1f75c8694a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:37.009496 30706 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:37.013945 30706 tablet_bootstrap.cc:654] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.015668 30706 log.cc:824] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:37.024919 30708 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:37.028565 30708 tablet_bootstrap.cc:654] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.029460 30706 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:37.029634 30708 log.cc:824] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:37.030009 30706 ts_tablet_manager.cc:1367] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.009s
I20240501 08:18:37.031086 30705 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:37.033290 30708 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:37.033558 30708 ts_tablet_manager.cc:1367] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240501 08:18:37.033898 30705 tablet_bootstrap.cc:654] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.034860 30705 log.cc:824] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:37.037156 30705 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:37.037384 30705 ts_tablet_manager.cc:1367] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.005s
I20240501 08:18:37.038933 30706 raft_consensus.cc:348] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.039341 30706 raft_consensus.cc:374] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.039434 30706 raft_consensus.cc:729] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.040618 30706 consensus_queue.cc:260] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.042060 30708 raft_consensus.cc:348] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.042438 30708 raft_consensus.cc:374] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.042536 30708 raft_consensus.cc:729] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.043012 30706 ts_tablet_manager.cc:1398] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.007s
I20240501 08:18:37.043555 30706 tablet_bootstrap.cc:492] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:37.043188 30708 consensus_queue.cc:260] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.044977 30705 raft_consensus.cc:348] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.045364 30706 tablet_bootstrap.cc:654] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.045447 30705 raft_consensus.cc:374] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.045559 30705 raft_consensus.cc:729] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.046303 30705 consensus_queue.cc:260] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.048979 30711 raft_consensus.cc:482] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:37.049937 30711 raft_consensus.cc:504] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.051173 30706 tablet_bootstrap.cc:492] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:37.051405 30706 ts_tablet_manager.cc:1367] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240501 08:18:37.052759 30706 raft_consensus.cc:348] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.053000 30706 raft_consensus.cc:374] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.053114 30706 raft_consensus.cc:729] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.053483 30706 consensus_queue.cc:260] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.054136 30706 ts_tablet_manager.cc:1398] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:37.054486 30706 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:37.059707 30706 tablet_bootstrap.cc:654] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.060073 30708 ts_tablet_manager.cc:1398] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.016s
I20240501 08:18:37.060531 30708 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:37.062681 30708 tablet_bootstrap.cc:654] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.071686 30708 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:37.071838 30708 ts_tablet_manager.cc:1367] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.008s
I20240501 08:18:37.072813 30708 raft_consensus.cc:348] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.073029 30708 raft_consensus.cc:374] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.073102 30708 raft_consensus.cc:729] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.073362 30708 consensus_queue.cc:260] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.073918 30708 ts_tablet_manager.cc:1398] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:37.074149 30708 tablet_bootstrap.cc:492] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:37.075589 30708 tablet_bootstrap.cc:654] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
W20240501 08:18:37.076788 30289 tablet.cc:2369] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:18:37.081756 30708 tablet_bootstrap.cc:492] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:37.081990 30708 ts_tablet_manager.cc:1367] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240501 08:18:37.082943 30708 raft_consensus.cc:348] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.083252 30708 raft_consensus.cc:374] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.083340 30708 raft_consensus.cc:729] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.083724 30708 consensus_queue.cc:260] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.084712 30708 ts_tablet_manager.cc:1398] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:37.084753 30713 raft_consensus.cc:482] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:37.084993 30713 raft_consensus.cc:504] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.087061 30713 leader_election.cc:288] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:37.117372 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a85f5b4f62343a4812d8c1f75c8694a" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890" is_pre_election: true
I20240501 08:18:37.118194 30374 raft_consensus.cc:2454] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:37.119102 30046 leader_election.cc:302] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:37.119604 30713 raft_consensus.cc:2781] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:37.119746 30713 raft_consensus.cc:482] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:37.119876 30713 raft_consensus.cc:3037] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.120342 30711 leader_election.cc:288] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:37.123613 30713 raft_consensus.cc:504] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.124475 30713 leader_election.cc:288] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Requested vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:37.128242 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a85f5b4f62343a4812d8c1f75c8694a" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:37.128526 30374 raft_consensus.cc:3037] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:18:37.131794 30157 tablet.cc:2369] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:18:37.133102 30374 raft_consensus.cc:2454] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:37.133749 30046 leader_election.cc:302] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:37.134447 30713 raft_consensus.cc:2781] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:37.141563 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a85f5b4f62343a4812d8c1f75c8694a" candidate_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de" is_pre_election: true
I20240501 08:18:37.149817 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a85f5b4f62343a4812d8c1f75c8694a" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:37.150269 30243 raft_consensus.cc:2454] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:37.150661 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a85f5b4f62343a4812d8c1f75c8694a" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:37.150913 30242 raft_consensus.cc:3037] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.152938 30713 raft_consensus.cc:686] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Becoming Leader. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Running, Role: LEADER
I20240501 08:18:37.153501 30713 consensus_queue.cc:237] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.153734 30705 ts_tablet_manager.cc:1398] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.116s	user 0.051s	sys 0.042s
I20240501 08:18:37.154317 30705 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:37.155673 30242 raft_consensus.cc:2454] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:37.155877 30705 tablet_bootstrap.cc:654] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.159766 30706 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:37.159955 30706 ts_tablet_manager.cc:1367] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.106s	user 0.017s	sys 0.042s
I20240501 08:18:37.160945 30706 raft_consensus.cc:348] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.161244 30706 raft_consensus.cc:374] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.161329 30706 raft_consensus.cc:729] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.161664 30706 consensus_queue.cc:260] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.162391 30706 ts_tablet_manager.cc:1398] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240501 08:18:37.164460 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a85f5b4f62343a4812d8c1f75c8694a" candidate_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890" is_pre_election: true
I20240501 08:18:37.164774 30374 raft_consensus.cc:2379] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7ef43f26720f4f4bbba3ebd7be22c26a in current term 1: Already voted for candidate 23d1b1dde90249cf82ab29f1b44795de in this term.
I20240501 08:18:37.165050 30705 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:37.165169 30705 ts_tablet_manager.cc:1367] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.004s
I20240501 08:18:37.166088 30705 raft_consensus.cc:348] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.166313 30705 raft_consensus.cc:374] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.166239 30178 leader_election.cc:302] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890
I20240501 08:18:37.166399 30705 raft_consensus.cc:729] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.166621 30705 consensus_queue.cc:260] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.167150 30705 ts_tablet_manager.cc:1398] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:18:37.167421 30705 tablet_bootstrap.cc:492] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:37.167448 29969 catalog_manager.cc:5484] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: term changed from 0 to 1, leader changed from <none> to 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129). New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:37.168241 30712 raft_consensus.cc:2726] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:18:37.169035 30705 tablet_bootstrap.cc:654] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:37.172142 30705 tablet_bootstrap.cc:492] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:37.172318 30705 ts_tablet_manager.cc:1367] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240501 08:18:37.172896 30705 raft_consensus.cc:348] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.173085 30705 raft_consensus.cc:374] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:37.173163 30705 raft_consensus.cc:729] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:37.173393 30705 consensus_queue.cc:260] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.173910 30705 ts_tablet_manager.cc:1398] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:18:37.224097 30715 raft_consensus.cc:482] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:37.224280 30715 raft_consensus.cc:504] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.225754 30715 leader_election.cc:288] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:37.232092 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de" is_pre_election: true
I20240501 08:18:37.232095 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:37.232342 30242 raft_consensus.cc:2454] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 0.
I20240501 08:18:37.232363 30111 raft_consensus.cc:2454] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 0.
I20240501 08:18:37.232774 30310 leader_election.cc:302] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7b5a0a22114297942fe102e2cbd890, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:37.233142 30715 raft_consensus.cc:2781] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:37.233247 30715 raft_consensus.cc:482] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:37.233330 30715 raft_consensus.cc:3037] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.235206 30715 raft_consensus.cc:504] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.235810 30715 leader_election.cc:288] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Requested vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:37.236179 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
I20240501 08:18:37.236371 30111 raft_consensus.cc:3037] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.236312 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:37.236492 30242 raft_consensus.cc:3037] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.238174 30111 raft_consensus.cc:2454] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:37.238260 30242 raft_consensus.cc:2454] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:37.238572 30309 leader_election.cc:302] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:37.238833 30715 raft_consensus.cc:2781] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:37.239215 30715 raft_consensus.cc:686] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Running, Role: LEADER
I20240501 08:18:37.239655 30715 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } }
I20240501 08:18:37.243881 29968 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131). New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } }
W20240501 08:18:37.267103 30420 tablet.cc:2369] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:18:37.318425 30713 raft_consensus.cc:482] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:37.318648 30713 raft_consensus.cc:504] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.319280 30713 leader_election.cc:288] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:37.319818 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844293e1c98e41b1bcb772284d7d7075" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:37.319845 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844293e1c98e41b1bcb772284d7d7075" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890" is_pre_election: true
I20240501 08:18:37.320032 30242 raft_consensus.cc:2454] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:37.320081 30374 raft_consensus.cc:2454] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:37.320498 30046 leader_election.cc:302] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:37.320776 30713 raft_consensus.cc:2781] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:37.320852 30713 raft_consensus.cc:482] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:37.320951 30713 raft_consensus.cc:3037] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.322741 30713 raft_consensus.cc:504] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.323274 30713 leader_election.cc:288] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Requested vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:37.323688 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844293e1c98e41b1bcb772284d7d7075" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:37.323779 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844293e1c98e41b1bcb772284d7d7075" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:37.323899 30242 raft_consensus.cc:3037] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.324007 30374 raft_consensus.cc:3037] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:37.326702 30242 raft_consensus.cc:2454] T 844293e1c98e41b1bcb772284d7d7075 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:37.326759 30374 raft_consensus.cc:2454] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:37.327139 30047 leader_election.cc:302] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:37.327409 30713 raft_consensus.cc:2781] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:37.327549 30713 raft_consensus.cc:686] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Becoming Leader. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Running, Role: LEADER
I20240501 08:18:37.327842 30713 consensus_queue.cc:237] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:37.330395 29968 catalog_manager.cc:5484] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: term changed from 0 to 1, leader changed from <none> to 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129). New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:37.549623 30713 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:37.557410 30713 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:18:37.680871 30733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:37.681171 30733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:37.688380 30733 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:18:37.722293 30715 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:37.727216 30715 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 08:18:37.819708 30713 consensus_queue.cc:1035] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:37.823807 30374 raft_consensus.cc:1261] T 844293e1c98e41b1bcb772284d7d7075 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:18:37.824381 30713 consensus_queue.cc:1035] T 844293e1c98e41b1bcb772284d7d7075 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 209.747 ms
  time per row: 0.104874 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240501 08:18:38.983688 29968 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33194:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 08:18:38.984473 29968 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:18:38.992655 30221 tablet_service.cc:1467] Processing CreateTablet for tablet cdc5670f722b437198eaf65666e35377 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:38.993127 30221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc5670f722b437198eaf65666e35377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:38.993222 30090 tablet_service.cc:1467] Processing CreateTablet for tablet d68d2a57cdee4440bb751927476725c5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:38.993613 30090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68d2a57cdee4440bb751927476725c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:38.993973 30352 tablet_service.cc:1467] Processing CreateTablet for tablet cdc5670f722b437198eaf65666e35377 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:38.994282 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc5670f722b437198eaf65666e35377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:38.996518 30354 tablet_service.cc:1467] Processing CreateTablet for tablet d68d2a57cdee4440bb751927476725c5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:38.996871 30354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68d2a57cdee4440bb751927476725c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:38.999472 30091 tablet_service.cc:1467] Processing CreateTablet for tablet cdc5670f722b437198eaf65666e35377 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:38.999927 30091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc5670f722b437198eaf65666e35377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.000203 30089 tablet_service.cc:1467] Processing CreateTablet for tablet 9c596e34a1384dcf897b53b01bd01447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.000571 30089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c596e34a1384dcf897b53b01bd01447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.004525 30222 tablet_service.cc:1467] Processing CreateTablet for tablet d68d2a57cdee4440bb751927476725c5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.005007 30222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68d2a57cdee4440bb751927476725c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.005836 30353 tablet_service.cc:1467] Processing CreateTablet for tablet 9c596e34a1384dcf897b53b01bd01447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.006235 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c596e34a1384dcf897b53b01bd01447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.008558 30223 tablet_service.cc:1467] Processing CreateTablet for tablet 9c596e34a1384dcf897b53b01bd01447 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.009022 30223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c596e34a1384dcf897b53b01bd01447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.010282 30803 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:39.012593 30803 tablet_bootstrap.cc:654] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.016201 30802 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:39.016894 30803 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:39.017043 30803 ts_tablet_manager.cc:1367] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240501 08:18:39.017846 30802 tablet_bootstrap.cc:654] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.017745 30803 raft_consensus.cc:348] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.018167 30803 raft_consensus.cc:374] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.018272 30803 raft_consensus.cc:729] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.018841 30803 consensus_queue.cc:260] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.019533 30803 ts_tablet_manager.cc:1398] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:39.020072 30803 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:39.023087 30803 tablet_bootstrap.cc:654] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.025035 30803 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:39.025233 30803 ts_tablet_manager.cc:1367] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:18:39.026067 30801 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:39.025966 30803 raft_consensus.cc:348] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.026245 30803 raft_consensus.cc:374] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.026367 30803 raft_consensus.cc:729] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.026677 30803 consensus_queue.cc:260] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.027808 30801 tablet_bootstrap.cc:654] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.029435 30803 ts_tablet_manager.cc:1398] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240501 08:18:39.030174 30803 tablet_bootstrap.cc:492] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:39.030225 30802 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:39.030392 30802 ts_tablet_manager.cc:1367] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.007s
I20240501 08:18:39.030929 30801 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:39.031073 30801 ts_tablet_manager.cc:1367] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240501 08:18:39.031463 30802 raft_consensus.cc:348] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.031633 30802 raft_consensus.cc:374] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.031725 30802 raft_consensus.cc:729] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.031807 30801 raft_consensus.cc:348] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.032013 30801 raft_consensus.cc:374] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.031946 30802 consensus_queue.cc:260] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.032160 30803 tablet_bootstrap.cc:654] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.032150 30801 raft_consensus.cc:729] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.032406 30801 consensus_queue.cc:260] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.033031 30802 ts_tablet_manager.cc:1398] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240501 08:18:39.033293 30802 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:39.034879 30802 tablet_bootstrap.cc:654] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.038776 30801 ts_tablet_manager.cc:1398] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20240501 08:18:39.039100 30801 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:39.039665 30803 tablet_bootstrap.cc:492] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:39.039871 30803 ts_tablet_manager.cc:1367] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240501 08:18:39.040701 30803 raft_consensus.cc:348] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.040846 30801 tablet_bootstrap.cc:654] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.041002 30803 raft_consensus.cc:374] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.041150 30803 raft_consensus.cc:729] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.041465 30803 consensus_queue.cc:260] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.042912 30803 ts_tablet_manager.cc:1398] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:39.043423 30802 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:39.043598 30802 ts_tablet_manager.cc:1367] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.009s
I20240501 08:18:39.044458 30802 raft_consensus.cc:348] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.044696 30802 raft_consensus.cc:374] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.044806 30802 raft_consensus.cc:729] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.045122 30802 consensus_queue.cc:260] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.045789 30802 ts_tablet_manager.cc:1398] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:39.046113 30802 tablet_bootstrap.cc:492] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:39.047317 30730 raft_consensus.cc:482] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:39.047461 30730 raft_consensus.cc:504] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.048341 30730 leader_election.cc:288] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:39.048841 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de" is_pre_election: true
I20240501 08:18:39.048959 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:39.049101 30111 raft_consensus.cc:2454] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 0.
I20240501 08:18:39.049363 30802 tablet_bootstrap.cc:654] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.050196 30309 leader_election.cc:302] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
W20240501 08:18:39.050434 30310 leader_election.cc:341] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:18:39.050629 30730 raft_consensus.cc:2781] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:39.050781 30730 raft_consensus.cc:482] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:39.050911 30730 raft_consensus.cc:3037] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.052523 30802 tablet_bootstrap.cc:492] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:39.052649 30801 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:39.052677 30802 ts_tablet_manager.cc:1367] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.003s
I20240501 08:18:39.052824 30801 ts_tablet_manager.cc:1367] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.005s
I20240501 08:18:39.053467 30802 raft_consensus.cc:348] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.053500 30801 raft_consensus.cc:348] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.053627 30802 raft_consensus.cc:374] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.053731 30802 raft_consensus.cc:729] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.053778 30801 raft_consensus.cc:374] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.053874 30801 raft_consensus.cc:729] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.053949 30802 consensus_queue.cc:260] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.054025 30730 raft_consensus.cc:504] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.054116 30801 consensus_queue.cc:260] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.054762 30801 ts_tablet_manager.cc:1398] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240501 08:18:39.055069 30801 tablet_bootstrap.cc:492] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:39.055454 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:39.055634 30242 raft_consensus.cc:3037] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.056777 30801 tablet_bootstrap.cc:654] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.058730 30242 raft_consensus.cc:2454] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:39.059724 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
I20240501 08:18:39.059991 30111 raft_consensus.cc:3037] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.060338 30801 tablet_bootstrap.cc:492] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:39.060511 30801 ts_tablet_manager.cc:1367] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240501 08:18:39.060983 30730 leader_election.cc:288] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Requested vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:39.061225 30730 raft_consensus.cc:482] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:39.061362 30730 raft_consensus.cc:504] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.061758 30801 raft_consensus.cc:348] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.062014 30801 raft_consensus.cc:374] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.062120 30801 raft_consensus.cc:729] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.062410 30801 consensus_queue.cc:260] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.062778 30111 raft_consensus.cc:2454] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:39.063588 30309 leader_election.cc:302] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:39.063979 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc5670f722b437198eaf65666e35377" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de" is_pre_election: true
I20240501 08:18:39.064106 30801 ts_tablet_manager.cc:1398] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240501 08:18:39.064252 30111 raft_consensus.cc:2454] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 0.
I20240501 08:18:39.064646 30309 leader_election.cc:302] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:39.064879 30730 leader_election.cc:288] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997)
I20240501 08:18:39.065084 30730 raft_consensus.cc:2781] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:39.065259 30730 raft_consensus.cc:686] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Running, Role: LEADER
I20240501 08:18:39.065565 30730 consensus_queue.cc:237] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.066526 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc5670f722b437198eaf65666e35377" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:39.066756 30242 raft_consensus.cc:2454] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 0.
I20240501 08:18:39.066972 30815 raft_consensus.cc:2781] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:39.067065 30815 raft_consensus.cc:482] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:39.067190 30815 raft_consensus.cc:3037] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.069381 30802 ts_tablet_manager.cc:1398] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.010s
I20240501 08:18:39.070657 29968 catalog_manager.cc:5484] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131). New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:39.071378 30815 raft_consensus.cc:504] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.072237 30815 leader_election.cc:288] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Requested vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997)
I20240501 08:18:39.072522 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc5670f722b437198eaf65666e35377" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
I20240501 08:18:39.072763 30111 raft_consensus.cc:3037] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.073141 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc5670f722b437198eaf65666e35377" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:39.073352 30242 raft_consensus.cc:3037] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.075495 30111 raft_consensus.cc:2454] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:39.075919 30309 leader_election.cc:302] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:39.075987 30242 raft_consensus.cc:2454] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:39.076228 30815 raft_consensus.cc:2781] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:39.076453 30815 raft_consensus.cc:686] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Running, Role: LEADER
I20240501 08:18:39.076756 30815 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.079569 29968 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131). New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:39.112336 30815 raft_consensus.cc:482] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:39.112514 30815 raft_consensus.cc:504] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.113106 30815 leader_election.cc:288] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:39.113626 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c596e34a1384dcf897b53b01bd01447" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de" is_pre_election: true
I20240501 08:18:39.113684 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c596e34a1384dcf897b53b01bd01447" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:39.113847 30111 raft_consensus.cc:2454] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 0.
I20240501 08:18:39.113950 30242 raft_consensus.cc:2454] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 0.
I20240501 08:18:39.114243 30309 leader_election.cc:302] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:39.114542 30815 raft_consensus.cc:2781] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:39.114642 30815 raft_consensus.cc:482] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:39.114711 30815 raft_consensus.cc:3037] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.116704 30815 raft_consensus.cc:504] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.117209 30815 leader_election.cc:288] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Requested vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:39.117703 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c596e34a1384dcf897b53b01bd01447" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
I20240501 08:18:39.117789 30242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c596e34a1384dcf897b53b01bd01447" candidate_uuid: "7e7b5a0a22114297942fe102e2cbd890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:39.117924 30111 raft_consensus.cc:3037] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.117986 30242 raft_consensus.cc:3037] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:39.120718 30111 raft_consensus.cc:2454] T 9c596e34a1384dcf897b53b01bd01447 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:39.120728 30242 raft_consensus.cc:2454] T 9c596e34a1384dcf897b53b01bd01447 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7b5a0a22114297942fe102e2cbd890 in term 1.
I20240501 08:18:39.121152 30309 leader_election.cc:302] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:39.121423 30815 raft_consensus.cc:2781] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:39.121552 30815 raft_consensus.cc:686] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Running, Role: LEADER
I20240501 08:18:39.121816 30815 consensus_queue.cc:237] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.124516 29968 catalog_manager.cc:5484] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131). New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } }
I20240501 08:18:39.467018 30815 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:39.473548 30815 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:39.480458 30815 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:39.485584 30730 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:18:39.487046 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:39.487463 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:39.495497 30819 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:18:39.554250 30730 consensus_queue.cc:1035] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:39.563791 30730 consensus_queue.cc:1035] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:18:39.739929 30309 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.2.248.129:34997, user_credentials={real_user=slave}} blocked reactor thread for 76103.8us

INSERT report
    rows total: 2000
    time total: 246.044 ms
  time per row: 0.123022 ms
I20240501 08:18:39.880424 29968 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33194:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240501 08:18:39.881139 29968 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 08:18:39.889283 30223 tablet_service.cc:1467] Processing CreateTablet for tablet b8b2b2af3fbf4a448c7fc431d8dde540 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.889381 30090 tablet_service.cc:1467] Processing CreateTablet for tablet b8b2b2af3fbf4a448c7fc431d8dde540 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.889803 30223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b2b2af3fbf4a448c7fc431d8dde540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.889863 30090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b2b2af3fbf4a448c7fc431d8dde540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.890733 30352 tablet_service.cc:1467] Processing CreateTablet for tablet b8b2b2af3fbf4a448c7fc431d8dde540 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.890983 30354 tablet_service.cc:1467] Processing CreateTablet for tablet 57bad390faac4b7f91c810137555662e (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.891178 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b2b2af3fbf4a448c7fc431d8dde540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.891389 30354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bad390faac4b7f91c810137555662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.897054 30089 tablet_service.cc:1467] Processing CreateTablet for tablet 57bad390faac4b7f91c810137555662e (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.897305 30221 tablet_service.cc:1467] Processing CreateTablet for tablet 57bad390faac4b7f91c810137555662e (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.897257 30091 tablet_service.cc:1467] Processing CreateTablet for tablet 63d2bb56fb3742f597302175ba63d9f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.897480 30089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bad390faac4b7f91c810137555662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.897652 30221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bad390faac4b7f91c810137555662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.897778 30091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d2bb56fb3742f597302175ba63d9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.901499 30881 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:39.903388 30881 tablet_bootstrap.cc:654] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.904031 30222 tablet_service.cc:1467] Processing CreateTablet for tablet 63d2bb56fb3742f597302175ba63d9f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.904414 30222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d2bb56fb3742f597302175ba63d9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.905805 30353 tablet_service.cc:1467] Processing CreateTablet for tablet 63d2bb56fb3742f597302175ba63d9f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:18:39.906320 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d2bb56fb3742f597302175ba63d9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:39.912712 30883 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:39.912853 30880 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:39.912951 30881 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:39.913082 30881 ts_tablet_manager.cc:1367] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240501 08:18:39.913910 30881 raft_consensus.cc:348] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.914129 30881 raft_consensus.cc:374] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.914225 30881 raft_consensus.cc:729] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.914557 30883 tablet_bootstrap.cc:654] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.914573 30880 tablet_bootstrap.cc:654] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.914726 30881 consensus_queue.cc:260] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.915418 30881 ts_tablet_manager.cc:1398] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:39.915724 30881 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:39.917217 30881 tablet_bootstrap.cc:654] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.925055 30880 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:39.925271 30880 ts_tablet_manager.cc:1367] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.005s
I20240501 08:18:39.925257 30881 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:39.925444 30881 ts_tablet_manager.cc:1367] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240501 08:18:39.926174 30881 raft_consensus.cc:348] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.926395 30881 raft_consensus.cc:374] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.926291 30880 raft_consensus.cc:348] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.926488 30881 raft_consensus.cc:729] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.926580 30880 raft_consensus.cc:374] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.926720 30880 raft_consensus.cc:729] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.926798 30881 consensus_queue.cc:260] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.927052 30880 consensus_queue.cc:260] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.927398 30881 ts_tablet_manager.cc:1398] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:39.928153 30880 ts_tablet_manager.cc:1398] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:39.928229 30881 tablet_bootstrap.cc:492] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a: Bootstrap starting.
I20240501 08:18:39.928418 30880 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:39.930102 30880 tablet_bootstrap.cc:654] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.930395 30881 tablet_bootstrap.cc:654] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.936890 30881 tablet_bootstrap.cc:492] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a: No bootstrap required, opened a new log
I20240501 08:18:39.937027 30881 ts_tablet_manager.cc:1367] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.004s
I20240501 08:18:39.937400 30883 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:39.937549 30883 ts_tablet_manager.cc:1367] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.025s	user 0.009s	sys 0.010s
I20240501 08:18:39.937718 30881 raft_consensus.cc:348] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.937871 30881 raft_consensus.cc:374] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.937973 30881 raft_consensus.cc:729] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.937983 30880 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:39.938148 30880 ts_tablet_manager.cc:1367] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.006s
I20240501 08:18:39.938184 30881 consensus_queue.cc:260] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.938357 30883 raft_consensus.cc:348] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.938561 30883 raft_consensus.cc:374] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.938643 30883 raft_consensus.cc:729] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.938665 30881 ts_tablet_manager.cc:1398] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:18:39.938869 30883 consensus_queue.cc:260] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.938935 30880 raft_consensus.cc:348] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.939186 30880 raft_consensus.cc:374] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.939253 30880 raft_consensus.cc:729] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.939529 30883 ts_tablet_manager.cc:1398] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:39.939477 30880 consensus_queue.cc:260] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.940130 30880 ts_tablet_manager.cc:1398] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240501 08:18:39.940495 30880 tablet_bootstrap.cc:492] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:39.941452 30883 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:39.942090 30880 tablet_bootstrap.cc:654] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.944725 30883 tablet_bootstrap.cc:654] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.946470 30883 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:39.946517 30880 tablet_bootstrap.cc:492] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de: No bootstrap required, opened a new log
I20240501 08:18:39.946617 30880 ts_tablet_manager.cc:1367] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.003s
I20240501 08:18:39.946621 30883 ts_tablet_manager.cc:1367] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.001s
I20240501 08:18:39.947266 30880 raft_consensus.cc:348] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.947290 30883 raft_consensus.cc:348] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.947454 30880 raft_consensus.cc:374] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.947486 30883 raft_consensus.cc:374] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.947553 30880 raft_consensus.cc:729] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.947556 30883 raft_consensus.cc:729] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.947778 30883 consensus_queue.cc:260] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.947764 30880 consensus_queue.cc:260] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.948274 30883 ts_tablet_manager.cc:1398] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:39.948565 30883 tablet_bootstrap.cc:492] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:39.948750 30880 ts_tablet_manager.cc:1398] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240501 08:18:39.949687 30883 tablet_bootstrap.cc:654] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890: Neither blocks nor log segments found. Creating new log.
I20240501 08:18:39.952421 30883 tablet_bootstrap.cc:492] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890: No bootstrap required, opened a new log
I20240501 08:18:39.952564 30883 ts_tablet_manager.cc:1367] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 08:18:39.953150 30883 raft_consensus.cc:348] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.953311 30883 raft_consensus.cc:374] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:18:39.953387 30883 raft_consensus.cc:729] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: FOLLOWER
I20240501 08:18:39.953568 30883 consensus_queue.cc:260] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:39.954010 30883 ts_tablet_manager.cc:1398] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:18:40.082351 30808 raft_consensus.cc:482] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:40.082532 30808 raft_consensus.cc:504] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.083313 30808 leader_election.cc:288] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:40.083765 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2b2af3fbf4a448c7fc431d8dde540" candidate_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de" is_pre_election: true
I20240501 08:18:40.083876 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2b2af3fbf4a448c7fc431d8dde540" candidate_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890" is_pre_election: true
I20240501 08:18:40.084045 30111 raft_consensus.cc:2454] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43f26720f4f4bbba3ebd7be22c26a in term 0.
I20240501 08:18:40.084146 30373 raft_consensus.cc:2454] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43f26720f4f4bbba3ebd7be22c26a in term 0.
I20240501 08:18:40.084563 30178 leader_election.cc:302] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:40.084820 30808 raft_consensus.cc:2781] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:40.084937 30808 raft_consensus.cc:482] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:40.085040 30808 raft_consensus.cc:3037] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.086910 30808 raft_consensus.cc:504] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.087419 30808 leader_election.cc:288] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [CANDIDATE]: Term 1 election: Requested vote from peers 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:40.087811 30111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2b2af3fbf4a448c7fc431d8dde540" candidate_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
I20240501 08:18:40.087900 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b2b2af3fbf4a448c7fc431d8dde540" candidate_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:40.087999 30111 raft_consensus.cc:3037] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.088099 30373 raft_consensus.cc:3037] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.089885 30111 raft_consensus.cc:2454] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43f26720f4f4bbba3ebd7be22c26a in term 1.
I20240501 08:18:40.090046 30373 raft_consensus.cc:2454] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43f26720f4f4bbba3ebd7be22c26a in term 1.
I20240501 08:18:40.090256 30178 leader_election.cc:302] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:40.090571 30808 raft_consensus.cc:2781] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:40.090976 30808 raft_consensus.cc:686] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef43f26720f4f4bbba3ebd7be22c26a, State: Running, Role: LEADER
I20240501 08:18:40.091464 30808 consensus_queue.cc:237] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.094798 29964 catalog_manager.cc:5484] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130). New cstate: current_term: 1 leader_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:40.143548 30724 raft_consensus.cc:482] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:40.143729 30724 raft_consensus.cc:504] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.144469 30724 leader_election.cc:288] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:40.144915 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2bb56fb3742f597302175ba63d9f6" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:40.144932 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2bb56fb3742f597302175ba63d9f6" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890" is_pre_election: true
I20240501 08:18:40.145136 30243 raft_consensus.cc:2454] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:40.145154 30373 raft_consensus.cc:2454] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:40.145566 30046 leader_election.cc:302] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:40.146015 30724 raft_consensus.cc:2781] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:40.146147 30724 raft_consensus.cc:482] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:40.146327 30724 raft_consensus.cc:3037] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.148936 30724 raft_consensus.cc:504] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.149516 30724 leader_election.cc:288] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Requested vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:40.149988 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2bb56fb3742f597302175ba63d9f6" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:40.150056 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d2bb56fb3742f597302175ba63d9f6" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:40.150184 30243 raft_consensus.cc:3037] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.150260 30373 raft_consensus.cc:3037] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.152024 30243 raft_consensus.cc:2454] T 63d2bb56fb3742f597302175ba63d9f6 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:40.152076 30373 raft_consensus.cc:2454] T 63d2bb56fb3742f597302175ba63d9f6 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:40.152457 30047 leader_election.cc:302] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:40.152719 30724 raft_consensus.cc:2781] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:40.153080 30724 raft_consensus.cc:686] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Becoming Leader. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Running, Role: LEADER
I20240501 08:18:40.153350 30724 consensus_queue.cc:237] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.155920 29964 catalog_manager.cc:5484] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: term changed from 0 to 1, leader changed from <none> to 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129). New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:40.247539 30724 raft_consensus.cc:482] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:18:40.247728 30724 raft_consensus.cc:504] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.248343 30724 leader_election.cc:288] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:40.248816 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bad390faac4b7f91c810137555662e" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890" is_pre_election: true
I20240501 08:18:40.248910 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bad390faac4b7f91c810137555662e" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" is_pre_election: true
I20240501 08:18:40.249050 30373 raft_consensus.cc:2454] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:40.249125 30243 raft_consensus.cc:2454] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 0.
I20240501 08:18:40.249505 30046 leader_election.cc:302] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890; no voters: 
I20240501 08:18:40.249856 30724 raft_consensus.cc:2781] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:18:40.249938 30724 raft_consensus.cc:482] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:18:40.250007 30724 raft_consensus.cc:3037] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.251789 30724 raft_consensus.cc:504] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.252478 30724 leader_election.cc:288] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Requested vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:40.252880 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bad390faac4b7f91c810137555662e" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:40.252969 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bad390faac4b7f91c810137555662e" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:40.253051 30243 raft_consensus.cc:3037] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.253167 30373 raft_consensus.cc:3037] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:18:40.254959 30243 raft_consensus.cc:2454] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:40.254966 30373 raft_consensus.cc:2454] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 1.
I20240501 08:18:40.255338 30046 leader_election.cc:302] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:40.255599 30724 raft_consensus.cc:2781] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:18:40.255745 30724 raft_consensus.cc:686] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Becoming Leader. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Running, Role: LEADER
I20240501 08:18:40.255985 30724 consensus_queue.cc:237] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } }
I20240501 08:18:40.258714 29964 catalog_manager.cc:5484] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: term changed from 0 to 1, leader changed from <none> to 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129). New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:40.610253 30899 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:40.622356 30899 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:18:40.629446 30897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:40.629896 30897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:40.637696 30897 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 08:18:40.695986 30724 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:40.705612 30724 consensus_queue.cc:1035] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:40.728761 30894 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:18:40.729198 30894 consensus_queue.cc:1035] T 63d2bb56fb3742f597302175ba63d9f6 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 223.011 ms
  time per row: 0.111506 ms
I20240501 08:18:40.999640  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.133:35633
--local_ip_for_outbound_sockets=127.2.248.133
--tserver_master_addrs=127.2.248.190:35115
--webserver_port=33629
--webserver_interface=127.2.248.133
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:18:41.323683 30961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:41.323994 30961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:41.324137 30961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:41.331552 30961 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:41.331661 30961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:41.331830 30961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.133
I20240501 08:18:41.340644 30961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.133:35633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.248.133
--webserver_port=33629
--tserver_master_addrs=127.2.248.190:35115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.133
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:41.341941 30961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:41.343544 30961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:41.355584 30968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:41.356063 30967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:41.356559 30970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:41.357194 30961 server_base.cc:1034] running on GCE node
I20240501 08:18:41.357745 30961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:41.358620 30961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:41.359823 30961 hybrid_clock.cc:648] HybridClock initialized: now 1714551521359775 us; error 36 us; skew 500 ppm
I20240501 08:18:41.360177 30961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:41.362612 30961 webserver.cc:460] Webserver started at http://127.2.248.133:33629/ using document root <none> and password file <none>
I20240501 08:18:41.363411 30961 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:41.363515 30961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:41.368516 30961 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240501 08:18:41.371064 30976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:41.372365 30961 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:41.372503 30961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "309c676e56d04218a47e557e6a834030"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:41.373271 30961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:41.393502 30961 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:41.394311 30961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:41.394585 30961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:41.395471 30961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:41.397006 30961 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:41.397074 30961 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:41.397193 30961 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:41.397244 30961 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:41.426611 30961 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.133:35633
I20240501 08:18:41.426647 31088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.133:35633 every 8 connection(s)
I20240501 08:18:41.428011 30961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:18:41.437595  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 30961
I20240501 08:18:41.438380  3042 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3vHYYh/build/asan/bin/kudu
/tmp/dist-test-task3vHYYh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.248.132:46099
--local_ip_for_outbound_sockets=127.2.248.132
--tserver_master_addrs=127.2.248.190:35115
--webserver_port=33097
--webserver_interface=127.2.248.132
--builtin_ntp_servers=127.2.248.148:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:18:41.439942 31089 heartbeater.cc:347] Connected to a master server at 127.2.248.190:35115
I20240501 08:18:41.440248 31089 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:41.441013 31089 heartbeater.cc:511] Master 127.2.248.190:35115 requested a full tablet report, sending...
I20240501 08:18:41.442857 29964 ts_manager.cc:194] Re-registered known tserver with Master: 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633)
I20240501 08:18:41.444339 29964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.133:50855
W20240501 08:18:41.767870 31093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:18:41.768220 31093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:18:41.768313 31093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:18:41.775738 31093 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:18:41.775872 31093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:18:41.775988 31093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.248.132
I20240501 08:18:41.784981 31093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.248.148:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.248.132:46099
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.248.132
--webserver_port=33097
--tserver_master_addrs=127.2.248.190:35115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.248.132
--log_dir=/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:18:41.786257 31093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:18:41.787811 31093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:18:41.798945 31098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:41.799415 31101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:18:41.799922 31099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:18:41.800580 31093 server_base.cc:1034] running on GCE node
I20240501 08:18:41.801173 31093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:18:41.802054 31093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:18:41.803265 31093 hybrid_clock.cc:648] HybridClock initialized: now 1714551521803208 us; error 37 us; skew 500 ppm
I20240501 08:18:41.803602 31093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:18:41.806385 31093 webserver.cc:460] Webserver started at http://127.2.248.132:33097/ using document root <none> and password file <none>
I20240501 08:18:41.807176 31093 fs_manager.cc:359] Metadata directory not provided
I20240501 08:18:41.807284 31093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:18:41.812208 31093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 08:18:41.814903 31107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:41.816290 31093 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240501 08:18:41.816430 31093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0492adfedf2f446dbfc904261c50475c"
format_stamp: "Formatted at 2024-05-01 08:18:32 on dist-test-slave-0pdg"
I20240501 08:18:41.817200 31093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:18:41.855270 31093 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:18:41.856034 31093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:18:41.856302 31093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:18:41.857177 31093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:18:41.858541 31093 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:18:41.858611 31093 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:41.858729 31093 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:18:41.858781 31093 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:18:41.889055 31093 rpc_server.cc:307] RPC server started. Bound to: 127.2.248.132:46099
I20240501 08:18:41.889099 31219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.248.132:46099 every 8 connection(s)
I20240501 08:18:41.890532 31093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:18:41.897214  3042 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3vHYYh/build/asan/bin/kudu as pid 31093
I20240501 08:18:41.904042 31220 heartbeater.cc:347] Connected to a master server at 127.2.248.190:35115
I20240501 08:18:41.904356 31220 heartbeater.cc:464] Registering TS with master...
I20240501 08:18:41.905118 31220 heartbeater.cc:511] Master 127.2.248.190:35115 requested a full tablet report, sending...
I20240501 08:18:41.906607 29964 ts_manager.cc:194] Re-registered known tserver with Master: 0492adfedf2f446dbfc904261c50475c (127.2.248.132:46099)
I20240501 08:18:41.907737 29964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.248.132:44985
I20240501 08:18:42.374059 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.376343 31024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.396807 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:42.402920 30091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.421124 30353 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.446708 31089 heartbeater.cc:503] Master 127.2.248.190:35115 was elected leader, sending a full tablet report...
I20240501 08:18:42.565531 30091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.567365 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.574685 30353 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.597555 31024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.619942 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:42.758721 30091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.758728 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.776105 31024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.785439 30353 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.798188 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:42.910310 31220 heartbeater.cc:503] Master 127.2.248.190:35115 was elected leader, sending a full tablet report...
I20240501 08:18:42.942502 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.949865 30091 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.960242 30353 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:18:42.967978 31024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.977185 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:43.043745 30241 consensus_queue.cc:237] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.046713 30374 raft_consensus.cc:1261] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43f26720f4f4bbba3ebd7be22c26a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.046792 30109 raft_consensus.cc:1261] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7ef43f26720f4f4bbba3ebd7be22c26a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.047362 31364 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.047789 30907 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.051417 30907 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.053117 30109 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.053320 30374 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
W20240501 08:18:43.055052 30179 consensus_peers.cc:485] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a -> Peer 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633): Couldn't send request to peer 309c676e56d04218a47e557e6a834030. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8b2b2af3fbf4a448c7fc431d8dde540. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:43.058936 29964 catalog_manager.cc:5484] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a reported cstate change: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:43.070907 30374 consensus_queue.cc:237] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.073832 30109 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.074527 31379 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.074959 30241 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.075706 31379 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.078075 30965 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.078596 30109 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.081300 30241 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
W20240501 08:18:43.085495 30310 consensus_peers.cc:485] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 -> Peer 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633): Couldn't send request to peer 309c676e56d04218a47e557e6a834030. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d68d2a57cdee4440bb751927476725c5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:43.085238 29964 catalog_manager.cc:5484] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:43.099227 30109 consensus_queue.cc:237] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.101924 30374 raft_consensus.cc:1261] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.102329 30241 raft_consensus.cc:1261] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.102456 31389 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.103350 31389 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.105252 31389 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.108821 30374 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
W20240501 08:18:43.109431 30047 consensus_peers.cc:485] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de -> Peer 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633): Couldn't send request to peer 309c676e56d04218a47e557e6a834030. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a85f5b4f62343a4812d8c1f75c8694a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:43.109628 30241 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.110898 29967 catalog_manager.cc:5484] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:43.125103 30374 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.127735 30241 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:18:43.127892 30310 consensus_peers.cc:485] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 -> Peer 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633): Couldn't send request to peer 309c676e56d04218a47e557e6a834030. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80b5f6bc3ac842ff931e03ae3094df55. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:43.127933 30109 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.128362 30965 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.128686 31379 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.131579 31379 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.132699 30109 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.132766 30241 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.136969 29967 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:43.151206 30374 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.153401 30241 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:43.153496 30109 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:18:43.153611 30310 consensus_peers.cc:485] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 -> Peer 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633): Couldn't send request to peer 309c676e56d04218a47e557e6a834030. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdc5670f722b437198eaf65666e35377. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:43.154011 31378 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.154280 31379 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:43.156968 30965 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.157619 30241 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.161224 30109 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:43.163105 29965 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:43.560632 31460 ts_tablet_manager.cc:902] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: Initiating tablet copy from peer 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997)
I20240501 08:18:43.562423 31460 tablet_copy_client.cc:308] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:34997
I20240501 08:18:43.568845 30131 tablet_copy_service.cc:140] P 23d1b1dde90249cf82ab29f1b44795de: Received BeginTabletCopySession request for tablet 2a85f5b4f62343a4812d8c1f75c8694a from peer 309c676e56d04218a47e557e6a834030 ({username='slave'} at 127.2.248.133:56675)
I20240501 08:18:43.569054 30131 tablet_copy_service.cc:161] P 23d1b1dde90249cf82ab29f1b44795de: Beginning new tablet copy session on tablet 2a85f5b4f62343a4812d8c1f75c8694a from peer 309c676e56d04218a47e557e6a834030 at {username='slave'} at 127.2.248.133:56675: session id = 309c676e56d04218a47e557e6a834030-2a85f5b4f62343a4812d8c1f75c8694a
I20240501 08:18:43.572325 30131 tablet_copy_source_session.cc:205] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:43.577486 31460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a85f5b4f62343a4812d8c1f75c8694a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:43.591234 31460 tablet_copy_client.cc:783] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:43.591579 31460 tablet_copy_client.cc:647] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:43.594111 31460 tablet_copy_client.cc:523] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:43.598335 31460 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: Bootstrap starting.
I20240501 08:18:43.608039 31460 log.cc:824] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: Log is configured to *not* fsync() on all Append() calls
I20240501 08:18:43.628067 31467 ts_tablet_manager.cc:902] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: Initiating tablet copy from peer 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:43.629871 31467 tablet_copy_client.cc:308] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34593
I20240501 08:18:43.637010 31472 ts_tablet_manager.cc:902] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: Initiating tablet copy from peer 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021)
I20240501 08:18:43.638046 30394 tablet_copy_service.cc:140] P 7e7b5a0a22114297942fe102e2cbd890: Received BeginTabletCopySession request for tablet 80b5f6bc3ac842ff931e03ae3094df55 from peer 309c676e56d04218a47e557e6a834030 ({username='slave'} at 127.2.248.133:51831)
I20240501 08:18:43.638264 30394 tablet_copy_service.cc:161] P 7e7b5a0a22114297942fe102e2cbd890: Beginning new tablet copy session on tablet 80b5f6bc3ac842ff931e03ae3094df55 from peer 309c676e56d04218a47e557e6a834030 at {username='slave'} at 127.2.248.133:51831: session id = 309c676e56d04218a47e557e6a834030-80b5f6bc3ac842ff931e03ae3094df55
I20240501 08:18:43.641029 31472 tablet_copy_client.cc:308] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.130:46021
I20240501 08:18:43.641321 30394 tablet_copy_source_session.cc:205] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:43.642954 31460 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:43.643162 31467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b5f6bc3ac842ff931e03ae3094df55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:43.643528 31460 tablet_bootstrap.cc:492] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: Bootstrap complete.
I20240501 08:18:43.644001 31460 ts_tablet_manager.cc:1367] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: Time spent bootstrapping tablet: real 0.046s	user 0.042s	sys 0.005s
I20240501 08:18:43.647348 30263 tablet_copy_service.cc:140] P 7ef43f26720f4f4bbba3ebd7be22c26a: Received BeginTabletCopySession request for tablet b8b2b2af3fbf4a448c7fc431d8dde540 from peer 309c676e56d04218a47e557e6a834030 ({username='slave'} at 127.2.248.133:44485)
I20240501 08:18:43.647565 30263 tablet_copy_service.cc:161] P 7ef43f26720f4f4bbba3ebd7be22c26a: Beginning new tablet copy session on tablet b8b2b2af3fbf4a448c7fc431d8dde540 from peer 309c676e56d04218a47e557e6a834030 at {username='slave'} at 127.2.248.133:44485: session id = 309c676e56d04218a47e557e6a834030-b8b2b2af3fbf4a448c7fc431d8dde540
I20240501 08:18:43.650023 31467 tablet_copy_client.cc:783] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:43.650295 31467 tablet_copy_client.cc:647] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:43.650921 31460 raft_consensus.cc:348] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.650954 30263 tablet_copy_source_session.cc:205] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:43.651378 31460 raft_consensus.cc:729] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 309c676e56d04218a47e557e6a834030, State: Initialized, Role: LEARNER
I20240501 08:18:43.651862 31460 consensus_queue.cc:260] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.652427 31467 tablet_copy_client.cc:523] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:43.653391 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b2b2af3fbf4a448c7fc431d8dde540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:43.655896 31467 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: Bootstrap starting.
I20240501 08:18:43.659914 31472 tablet_copy_client.cc:783] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:43.660225 31472 tablet_copy_client.cc:647] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:43.662520 31472 tablet_copy_client.cc:523] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:43.667498 31460 ts_tablet_manager.cc:1398] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030: Time spent starting tablet: real 0.023s	user 0.014s	sys 0.008s
I20240501 08:18:43.668569 30131 tablet_copy_service.cc:342] P 23d1b1dde90249cf82ab29f1b44795de: Request end of tablet copy session 309c676e56d04218a47e557e6a834030-2a85f5b4f62343a4812d8c1f75c8694a received from {username='slave'} at 127.2.248.133:56675
I20240501 08:18:43.668648 31479 ts_tablet_manager.cc:902] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: Initiating tablet copy from peer 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:43.668761 30131 tablet_copy_service.cc:433] P 23d1b1dde90249cf82ab29f1b44795de: ending tablet copy session 309c676e56d04218a47e557e6a834030-2a85f5b4f62343a4812d8c1f75c8694a on tablet 2a85f5b4f62343a4812d8c1f75c8694a with peer 309c676e56d04218a47e557e6a834030
I20240501 08:18:43.670083 31472 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: Bootstrap starting.
I20240501 08:18:43.690649 31460 ts_tablet_manager.cc:902] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: Initiating tablet copy from peer 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:43.695443 31460 tablet_copy_client.cc:308] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34593
I20240501 08:18:43.696246 30394 tablet_copy_service.cc:140] P 7e7b5a0a22114297942fe102e2cbd890: Received BeginTabletCopySession request for tablet cdc5670f722b437198eaf65666e35377 from peer 309c676e56d04218a47e557e6a834030 ({username='slave'} at 127.2.248.133:51831)
I20240501 08:18:43.696406 30394 tablet_copy_service.cc:161] P 7e7b5a0a22114297942fe102e2cbd890: Beginning new tablet copy session on tablet cdc5670f722b437198eaf65666e35377 from peer 309c676e56d04218a47e557e6a834030 at {username='slave'} at 127.2.248.133:51831: session id = 309c676e56d04218a47e557e6a834030-cdc5670f722b437198eaf65666e35377
I20240501 08:18:43.699234 30394 tablet_copy_source_session.cc:205] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:43.700547 31460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc5670f722b437198eaf65666e35377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:43.705204 31479 tablet_copy_client.cc:308] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34593
I20240501 08:18:43.706094 30394 tablet_copy_service.cc:140] P 7e7b5a0a22114297942fe102e2cbd890: Received BeginTabletCopySession request for tablet d68d2a57cdee4440bb751927476725c5 from peer 309c676e56d04218a47e557e6a834030 ({username='slave'} at 127.2.248.133:51831)
I20240501 08:18:43.706272 30394 tablet_copy_service.cc:161] P 7e7b5a0a22114297942fe102e2cbd890: Beginning new tablet copy session on tablet d68d2a57cdee4440bb751927476725c5 from peer 309c676e56d04218a47e557e6a834030 at {username='slave'} at 127.2.248.133:51831: session id = 309c676e56d04218a47e557e6a834030-d68d2a57cdee4440bb751927476725c5
I20240501 08:18:43.707530 31460 tablet_copy_client.cc:783] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:43.707806 31460 tablet_copy_client.cc:647] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:43.709465 31460 tablet_copy_client.cc:523] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:43.710449 30394 tablet_copy_source_session.cc:205] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:43.712177 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68d2a57cdee4440bb751927476725c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:43.716077 31460 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: Bootstrap starting.
I20240501 08:18:43.721793 31479 tablet_copy_client.cc:783] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:43.722036 31479 tablet_copy_client.cc:647] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:43.723660 31479 tablet_copy_client.cc:523] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:43.727031 31479 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: Bootstrap starting.
I20240501 08:18:43.734957 31467 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:43.735464 31467 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: Bootstrap complete.
I20240501 08:18:43.735796 31467 ts_tablet_manager.cc:1367] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: Time spent bootstrapping tablet: real 0.080s	user 0.057s	sys 0.006s
I20240501 08:18:43.737165 31467 raft_consensus.cc:348] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.737502 31467 raft_consensus.cc:729] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 309c676e56d04218a47e557e6a834030, State: Initialized, Role: LEARNER
I20240501 08:18:43.737780 31467 consensus_queue.cc:260] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.741232 31467 ts_tablet_manager.cc:1398] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:18:43.743734 31472 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:43.744253 31472 tablet_bootstrap.cc:492] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: Bootstrap complete.
I20240501 08:18:43.744630 31472 ts_tablet_manager.cc:1367] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: Time spent bootstrapping tablet: real 0.075s	user 0.048s	sys 0.007s
I20240501 08:18:43.745205 30394 tablet_copy_service.cc:342] P 7e7b5a0a22114297942fe102e2cbd890: Request end of tablet copy session 309c676e56d04218a47e557e6a834030-80b5f6bc3ac842ff931e03ae3094df55 received from {username='slave'} at 127.2.248.133:51831
I20240501 08:18:43.745416 30394 tablet_copy_service.cc:433] P 7e7b5a0a22114297942fe102e2cbd890: ending tablet copy session 309c676e56d04218a47e557e6a834030-80b5f6bc3ac842ff931e03ae3094df55 on tablet 80b5f6bc3ac842ff931e03ae3094df55 with peer 309c676e56d04218a47e557e6a834030
I20240501 08:18:43.745616 31472 raft_consensus.cc:348] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.745864 31472 raft_consensus.cc:729] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 309c676e56d04218a47e557e6a834030, State: Initialized, Role: LEARNER
I20240501 08:18:43.746088 31472 consensus_queue.cc:260] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.749424 31472 ts_tablet_manager.cc:1398] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240501 08:18:43.752372 30263 tablet_copy_service.cc:342] P 7ef43f26720f4f4bbba3ebd7be22c26a: Request end of tablet copy session 309c676e56d04218a47e557e6a834030-b8b2b2af3fbf4a448c7fc431d8dde540 received from {username='slave'} at 127.2.248.133:44485
I20240501 08:18:43.752539 30263 tablet_copy_service.cc:433] P 7ef43f26720f4f4bbba3ebd7be22c26a: ending tablet copy session 309c676e56d04218a47e557e6a834030-b8b2b2af3fbf4a448c7fc431d8dde540 on tablet b8b2b2af3fbf4a448c7fc431d8dde540 with peer 309c676e56d04218a47e557e6a834030
I20240501 08:18:43.785153 31479 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:43.785459 31460 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:43.785634 31479 tablet_bootstrap.cc:492] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: Bootstrap complete.
I20240501 08:18:43.785923 31479 ts_tablet_manager.cc:1367] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: Time spent bootstrapping tablet: real 0.059s	user 0.042s	sys 0.008s
I20240501 08:18:43.786254 31460 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: Bootstrap complete.
I20240501 08:18:43.786619 31460 ts_tablet_manager.cc:1367] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: Time spent bootstrapping tablet: real 0.071s	user 0.044s	sys 0.007s
I20240501 08:18:43.786713 31479 raft_consensus.cc:348] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.786947 31479 raft_consensus.cc:729] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 309c676e56d04218a47e557e6a834030, State: Initialized, Role: LEARNER
I20240501 08:18:43.787153 31479 consensus_queue.cc:260] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.787752 31479 ts_tablet_manager.cc:1398] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240501 08:18:43.787624 31460 raft_consensus.cc:348] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.788079 31460 raft_consensus.cc:729] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 309c676e56d04218a47e557e6a834030, State: Initialized, Role: LEARNER
I20240501 08:18:43.788283 31460 consensus_queue.cc:260] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:43.788473 30394 tablet_copy_service.cc:342] P 7e7b5a0a22114297942fe102e2cbd890: Request end of tablet copy session 309c676e56d04218a47e557e6a834030-d68d2a57cdee4440bb751927476725c5 received from {username='slave'} at 127.2.248.133:51831
I20240501 08:18:43.788614 30394 tablet_copy_service.cc:433] P 7e7b5a0a22114297942fe102e2cbd890: ending tablet copy session 309c676e56d04218a47e557e6a834030-d68d2a57cdee4440bb751927476725c5 on tablet d68d2a57cdee4440bb751927476725c5 with peer 309c676e56d04218a47e557e6a834030
I20240501 08:18:43.789803 31460 ts_tablet_manager.cc:1398] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240501 08:18:43.790426 30394 tablet_copy_service.cc:342] P 7e7b5a0a22114297942fe102e2cbd890: Request end of tablet copy session 309c676e56d04218a47e557e6a834030-cdc5670f722b437198eaf65666e35377 received from {username='slave'} at 127.2.248.133:51831
I20240501 08:18:43.790553 30394 tablet_copy_service.cc:433] P 7e7b5a0a22114297942fe102e2cbd890: ending tablet copy session 309c676e56d04218a47e557e6a834030-cdc5670f722b437198eaf65666e35377 on tablet cdc5670f722b437198eaf65666e35377 with peer 309c676e56d04218a47e557e6a834030
I20240501 08:18:44.104557 31044 raft_consensus.cc:1203] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:44.122644 31044 raft_consensus.cc:1203] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:44.137189 31044 raft_consensus.cc:1203] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:44.180374 31044 raft_consensus.cc:1203] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:44.230690 31044 raft_consensus.cc:1203] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:44.520195 31542 raft_consensus.cc:1050] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: attempting to promote NON_VOTER 309c676e56d04218a47e557e6a834030 to VOTER
I20240501 08:18:44.521131 31542 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.523530 30241 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.523578 30109 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.523667 31044 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.524268 30965 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.524884 30965 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.525352 30965 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.538455 31541 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.542837 30243 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.543071 30109 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.546736 29964 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:44.548069 31044 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.557169 30374 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.558878 31044 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:44.559047 30243 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:44.559623 31542 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:44.559942 31588 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:44.562885 30965 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.563310 31044 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.563228 30243 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.566556 29956 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80b5f6bc3ac842ff931e03ae3094df55 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:44.567950 29965 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 reported cstate change: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.571888 30091 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b5f6bc3ac842ff931e03ae3094df55 with delete_type TABLET_DATA_TOMBSTONED (TS 23d1b1dde90249cf82ab29f1b44795de not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46596
I20240501 08:18:44.577085 31606 tablet_replica.cc:330] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: stopping tablet replica
I20240501 08:18:44.577484 31606 raft_consensus.cc:2229] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:44.577934 31606 raft_consensus.cc:2258] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:44.579267 31606 ts_tablet_manager.cc:1875] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:44.583631 31606 ts_tablet_manager.cc:1888] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:44.583752 31606 log.cc:1196] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/80b5f6bc3ac842ff931e03ae3094df55
I20240501 08:18:44.584666 29956 catalog_manager.cc:4830] TS 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997): tablet 80b5f6bc3ac842ff931e03ae3094df55 (table rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]) successfully deleted
I20240501 08:18:44.623620 31544 raft_consensus.cc:1050] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de: attempting to promote NON_VOTER 309c676e56d04218a47e557e6a834030 to VOTER
I20240501 08:18:44.624466 31544 consensus_queue.cc:237] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.626504 31044 raft_consensus.cc:1261] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.626749 30374 raft_consensus.cc:1261] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.626776 30243 raft_consensus.cc:1261] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.626973 31543 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.627291 31544 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.627552 31608 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.632232 31542 raft_consensus.cc:1050] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: attempting to promote NON_VOTER 309c676e56d04218a47e557e6a834030 to VOTER
I20240501 08:18:44.633332 31542 consensus_queue.cc:237] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.636023 31546 raft_consensus.cc:1050] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a: attempting to promote NON_VOTER 309c676e56d04218a47e557e6a834030 to VOTER
I20240501 08:18:44.637323 31546 consensus_queue.cc:237] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.640806 30373 raft_consensus.cc:1261] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43f26720f4f4bbba3ebd7be22c26a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.641623 31546 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.642144 30109 raft_consensus.cc:1261] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7ef43f26720f4f4bbba3ebd7be22c26a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.643738 31546 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.644569 31043 raft_consensus.cc:1261] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Refusing update from remote peer 7ef43f26720f4f4bbba3ebd7be22c26a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.645145 31546 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.647579 31608 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.648335 31044 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.652684 29964 catalog_manager.cc:5484] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 reported cstate change: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.652840 30374 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.656502 30243 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.658684 30111 consensus_queue.cc:237] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.660647 30374 raft_consensus.cc:1261] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:44.661172 31608 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:44.665797 31044 raft_consensus.cc:1261] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:44.667929 31544 consensus_queue.cc:1035] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:44.668869 31543 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.669396 30372 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.674095 29956 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a85f5b4f62343a4812d8c1f75c8694a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:44.674937 29964 catalog_manager.cc:5484] T 2a85f5b4f62343a4812d8c1f75c8694a P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:44.675840 31044 raft_consensus.cc:2932] T 2a85f5b4f62343a4812d8c1f75c8694a P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.676592 31043 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.676604 30243 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.677239 31541 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.677798 31541 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.678874 30223 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a85f5b4f62343a4812d8c1f75c8694a with delete_type TABLET_DATA_TOMBSTONED (TS 7ef43f26720f4f4bbba3ebd7be22c26a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50852
I20240501 08:18:44.679320 30110 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.680637 30965 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.682504 31619 tablet_replica.cc:330] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: stopping tablet replica
I20240501 08:18:44.682482 31546 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.683030 31619 raft_consensus.cc:2229] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:44.683367 30373 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.684229 31619 raft_consensus.cc:2258] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:44.685120 30111 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.689462 29967 catalog_manager.cc:5484] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.692627 30965 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.694331 31619 ts_tablet_manager.cc:1875] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:44.695417 30243 consensus_queue.cc:237] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.696102 31043 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.697583 30373 raft_consensus.cc:1261] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43f26720f4f4bbba3ebd7be22c26a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:44.698076 31546 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:44.699504 31042 raft_consensus.cc:1261] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43f26720f4f4bbba3ebd7be22c26a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:44.700237 30907 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.700882 30373 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.701709 31546 consensus_queue.cc:1035] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:44.702004 31043 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.703038 31042 raft_consensus.cc:2932] T b8b2b2af3fbf4a448c7fc431d8dde540 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.703812 29957 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8b2b2af3fbf4a448c7fc431d8dde540 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:44.705858 29967 catalog_manager.cc:5484] T b8b2b2af3fbf4a448c7fc431d8dde540 P 7ef43f26720f4f4bbba3ebd7be22c26a reported cstate change: config changed from index 5 to 6, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New cstate: current_term: 1 leader_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:44.707158 31619 ts_tablet_manager.cc:1888] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:44.707373 31619 log.cc:1196] T 2a85f5b4f62343a4812d8c1f75c8694a P 7ef43f26720f4f4bbba3ebd7be22c26a: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/2a85f5b4f62343a4812d8c1f75c8694a
I20240501 08:18:44.708487 29957 catalog_manager.cc:4830] TS 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021): tablet 2a85f5b4f62343a4812d8c1f75c8694a (table rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]) successfully deleted
I20240501 08:18:44.708257 29963 catalog_manager.cc:5484] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 reported cstate change: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.709084 30243 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.710028 30109 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.711547 30091 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b2b2af3fbf4a448c7fc431d8dde540 with delete_type TABLET_DATA_TOMBSTONED (TS 23d1b1dde90249cf82ab29f1b44795de not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46596
I20240501 08:18:44.712980 31606 tablet_replica.cc:330] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: stopping tablet replica
I20240501 08:18:44.713177 31606 raft_consensus.cc:2229] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:44.713373 31606 raft_consensus.cc:2258] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:44.714871 31606 ts_tablet_manager.cc:1875] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:44.719357 31606 ts_tablet_manager.cc:1888] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:44.719449 31606 log.cc:1196] T b8b2b2af3fbf4a448c7fc431d8dde540 P 23d1b1dde90249cf82ab29f1b44795de: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8b2b2af3fbf4a448c7fc431d8dde540
I20240501 08:18:44.720158 29956 catalog_manager.cc:4830] TS 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997): tablet b8b2b2af3fbf4a448c7fc431d8dde540 (table rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]) successfully deleted
I20240501 08:18:44.725399 31618 raft_consensus.cc:1050] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: attempting to promote NON_VOTER 309c676e56d04218a47e557e6a834030 to VOTER
I20240501 08:18:44.726025 31618 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:44.728183 30109 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.728299 30243 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.728322 31043 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:44.728717 31588 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.728947 31542 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.729233 31618 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:44.732630 31588 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.733311 30243 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.733880 31043 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.737509 29964 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a reported cstate change: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.737237 30109 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:44.807950 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cdc5670f722b437198eaf65666e35377"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59916
I20240501 08:18:44.808243 30373 raft_consensus.cc:595] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:44.823751 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59936
I20240501 08:18:44.823946 30373 raft_consensus.cc:595] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:44.835817 30109 consensus_queue.cc:237] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:44.837944 30373 raft_consensus.cc:1261] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:18:44.838080 30243 raft_consensus.cc:1261] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:18:44.838181 30047 consensus_peers.cc:485] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de -> Peer 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633): Couldn't send request to peer 309c676e56d04218a47e557e6a834030. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57bad390faac4b7f91c810137555662e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:44.838455 31608 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:44.838733 31617 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:18:44.841377 31608 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:44.842538 30243 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:44.844467 30373 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:44.849211 29967 catalog_manager.cc:5484] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a reported cstate change: config changed from index -1 to 4, NON_VOTER 309c676e56d04218a47e557e6a834030 (127.2.248.133) added. New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } } }
I20240501 08:18:44.873100 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cdc5670f722b437198eaf65666e35377"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59968
I20240501 08:18:44.873281 30373 raft_consensus.cc:595] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:44.889220 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59984
I20240501 08:18:44.889443 30373 raft_consensus.cc:595] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:45.114252 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cdc5670f722b437198eaf65666e35377"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60006
I20240501 08:18:45.114446 30373 raft_consensus.cc:595] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:45.125859 31541 raft_consensus.cc:979] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: : Instructing follower 23d1b1dde90249cf82ab29f1b44795de to start an election
I20240501 08:18:45.126058 31541 raft_consensus.cc:1067] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Signalling peer 23d1b1dde90249cf82ab29f1b44795de to start an election
I20240501 08:18:45.127058 30109 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5"
dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
 from {username='slave'} at 127.2.248.131:51551
I20240501 08:18:45.127322 30109 raft_consensus.cc:482] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:45.127465 30109 raft_consensus.cc:3037] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.130445 30109 raft_consensus.cc:504] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:45.131837 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:45.132112 30243 raft_consensus.cc:3037] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.132196 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60026
I20240501 08:18:45.132401 30373 raft_consensus.cc:595] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:45.134173 30243 raft_consensus.cc:2454] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 2.
I20240501 08:18:45.135051 30109 leader_election.cc:288] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 2 election: Requested vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593), 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633)
I20240501 08:18:45.135378 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:45.135582 30373 raft_consensus.cc:3032] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:45.135658 30373 raft_consensus.cc:729] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Running, Role: LEADER
I20240501 08:18:45.135993 30373 consensus_queue.cc:260] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:45.136497 30373 raft_consensus.cc:3037] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.136438 31043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68d2a57cdee4440bb751927476725c5" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "309c676e56d04218a47e557e6a834030"
I20240501 08:18:45.136734 31043 raft_consensus.cc:3037] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.138432 30373 raft_consensus.cc:2454] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 2.
I20240501 08:18:45.138890 30046 leader_election.cc:302] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 7e7b5a0a22114297942fe102e2cbd890, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:45.139200 31608 raft_consensus.cc:2781] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:45.139348 31608 raft_consensus.cc:686] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 2 LEADER]: Becoming Leader. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Running, Role: LEADER
I20240501 08:18:45.139621 31608 consensus_queue.cc:237] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:45.139890 31043 raft_consensus.cc:2454] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 2.
I20240501 08:18:45.142275 29968 catalog_manager.cc:5484] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: term changed from 1 to 2, leader changed from 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) to 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129). New cstate: current_term: 2 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:45.169301 31541 raft_consensus.cc:979] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: : Instructing follower 309c676e56d04218a47e557e6a834030 to start an election
I20240501 08:18:45.169486 31541 raft_consensus.cc:1067] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Signalling peer 309c676e56d04218a47e557e6a834030 to start an election
I20240501 08:18:45.170295 31043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cdc5670f722b437198eaf65666e35377"
dest_uuid: "309c676e56d04218a47e557e6a834030"
 from {username='slave'} at 127.2.248.131:51281
I20240501 08:18:45.170573 31043 raft_consensus.cc:482] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:45.170689 31043 raft_consensus.cc:3037] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.172693 31043 raft_consensus.cc:504] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:45.174116 31043 leader_election.cc:288] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [CANDIDATE]: Term 2 election: Requested vote from peers 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997), 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:45.174602 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc5670f722b437198eaf65666e35377" candidate_uuid: "309c676e56d04218a47e557e6a834030" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
I20240501 08:18:45.174634 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc5670f722b437198eaf65666e35377" candidate_uuid: "309c676e56d04218a47e557e6a834030" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:45.174757 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc5670f722b437198eaf65666e35377" candidate_uuid: "309c676e56d04218a47e557e6a834030" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:45.174849 30243 raft_consensus.cc:3037] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.174860 30109 raft_consensus.cc:3037] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.174986 30373 raft_consensus.cc:3032] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:45.175068 30373 raft_consensus.cc:729] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Running, Role: LEADER
I20240501 08:18:45.175380 30373 consensus_queue.cc:260] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:45.175745 30373 raft_consensus.cc:3037] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:45.177628 30243 raft_consensus.cc:2454] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309c676e56d04218a47e557e6a834030 in term 2.
I20240501 08:18:45.177613 30109 raft_consensus.cc:2454] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309c676e56d04218a47e557e6a834030 in term 2.
I20240501 08:18:45.177695 30373 raft_consensus.cc:2454] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309c676e56d04218a47e557e6a834030 in term 2.
I20240501 08:18:45.178169 30979 leader_election.cc:302] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 309c676e56d04218a47e557e6a834030, 7e7b5a0a22114297942fe102e2cbd890, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:45.178529 31477 raft_consensus.cc:2781] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:45.178972 31477 raft_consensus.cc:686] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 2 LEADER]: Becoming Leader. State: Replica: 309c676e56d04218a47e557e6a834030, State: Running, Role: LEADER
I20240501 08:18:45.179477 31477 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:45.182667 29968 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 reported cstate change: term changed from 1 to 2, leader changed from 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) to 309c676e56d04218a47e557e6a834030 (127.2.248.133). New cstate: current_term: 2 leader_uuid: "309c676e56d04218a47e557e6a834030" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:45.282989 31723 ts_tablet_manager.cc:902] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: Initiating tablet copy from peer 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997)
I20240501 08:18:45.284628 31723 tablet_copy_client.cc:308] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.129:34997
I20240501 08:18:45.285343 30131 tablet_copy_service.cc:140] P 23d1b1dde90249cf82ab29f1b44795de: Received BeginTabletCopySession request for tablet 57bad390faac4b7f91c810137555662e from peer 309c676e56d04218a47e557e6a834030 ({username='slave'} at 127.2.248.133:56675)
I20240501 08:18:45.285498 30131 tablet_copy_service.cc:161] P 23d1b1dde90249cf82ab29f1b44795de: Beginning new tablet copy session on tablet 57bad390faac4b7f91c810137555662e from peer 309c676e56d04218a47e557e6a834030 at {username='slave'} at 127.2.248.133:56675: session id = 309c676e56d04218a47e557e6a834030-57bad390faac4b7f91c810137555662e
I20240501 08:18:45.287400 30131 tablet_copy_source_session.cc:205] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:45.288544 31723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bad390faac4b7f91c810137555662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:45.293044 31723 tablet_copy_client.cc:783] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:45.293267 31723 tablet_copy_client.cc:647] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:45.294701 31723 tablet_copy_client.cc:523] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:45.297060 31723 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: Bootstrap starting.
I20240501 08:18:45.327987 31723 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:45.328428 31723 tablet_bootstrap.cc:492] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: Bootstrap complete.
I20240501 08:18:45.328827 31723 ts_tablet_manager.cc:1367] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: Time spent bootstrapping tablet: real 0.032s	user 0.027s	sys 0.005s
I20240501 08:18:45.329650 31723 raft_consensus.cc:348] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:45.329887 31723 raft_consensus.cc:729] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 309c676e56d04218a47e557e6a834030, State: Initialized, Role: LEARNER
I20240501 08:18:45.330052 31723 consensus_queue.cc:260] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: NON_VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: true } }
I20240501 08:18:45.330468 31723 ts_tablet_manager.cc:1398] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240501 08:18:45.331276 30131 tablet_copy_service.cc:342] P 23d1b1dde90249cf82ab29f1b44795de: Request end of tablet copy session 309c676e56d04218a47e557e6a834030-57bad390faac4b7f91c810137555662e received from {username='slave'} at 127.2.248.133:56675
I20240501 08:18:45.331413 30131 tablet_copy_service.cc:433] P 23d1b1dde90249cf82ab29f1b44795de: ending tablet copy session 309c676e56d04218a47e557e6a834030-57bad390faac4b7f91c810137555662e on tablet 57bad390faac4b7f91c810137555662e with peer 309c676e56d04218a47e557e6a834030
I20240501 08:18:45.583024 30373 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:45.583631 31477 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:45.620942 30243 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:45.621562 31617 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:45.638343 30109 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:45.638927 31477 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:45.643044 30243 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:45.643656 31477 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:45.678887 31043 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:45.679438 31617 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:45.683511 30373 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:18:45.684791 31617 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:45.691141 30109 consensus_queue.cc:237] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:45.693027 31043 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:18:45.693035 30243 raft_consensus.cc:1261] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:18:45.693500 31608 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:45.693724 31617 consensus_queue.cc:1035] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:45.695690 31765 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 23d1b1dde90249cf82ab29f1b44795de [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:45.696307 31043 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:45.696251 30243 raft_consensus.cc:2932] T d68d2a57cdee4440bb751927476725c5 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:45.699494 29956 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d68d2a57cdee4440bb751927476725c5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:45.700435 29964 catalog_manager.cc:5484] T d68d2a57cdee4440bb751927476725c5 P 309c676e56d04218a47e557e6a834030 reported cstate change: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New cstate: current_term: 2 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:45.705163 30353 tablet_service.cc:1514] Processing DeleteTablet for tablet d68d2a57cdee4440bb751927476725c5 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7b5a0a22114297942fe102e2cbd890 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33992
I20240501 08:18:45.705948 31769 tablet_replica.cc:330] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: stopping tablet replica
I20240501 08:18:45.706321 31769 raft_consensus.cc:2229] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:45.706535 31769 raft_consensus.cc:2258] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:45.707741 31769 ts_tablet_manager.cc:1875] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:45.712178 31769 ts_tablet_manager.cc:1888] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:18:45.712296 31769 log.cc:1196] T d68d2a57cdee4440bb751927476725c5 P 7e7b5a0a22114297942fe102e2cbd890: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/d68d2a57cdee4440bb751927476725c5
I20240501 08:18:45.713040 29956 catalog_manager.cc:4830] TS 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593): tablet d68d2a57cdee4440bb751927476725c5 (table rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]) successfully deleted
I20240501 08:18:45.839697 31043 raft_consensus.cc:1203] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240501 08:18:46.131870 31043 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:46.133941 30243 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:18:46.134001 30109 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:18:46.134582 31477 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:46.135056 31720 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:46.137327 31477 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.137955 30109 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.138648 30243 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.141602 29957 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cdc5670f722b437198eaf65666e35377 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:46.142575 29968 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 reported cstate change: config changed from index 5 to 7, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New cstate: current_term: 2 leader_uuid: "309c676e56d04218a47e557e6a834030" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:46.146301 30353 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc5670f722b437198eaf65666e35377 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7b5a0a22114297942fe102e2cbd890 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33992
I20240501 08:18:46.146823 31769 tablet_replica.cc:330] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: stopping tablet replica
I20240501 08:18:46.147017 31769 raft_consensus.cc:2229] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:46.147187 31769 raft_consensus.cc:2258] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:46.147979 31769 ts_tablet_manager.cc:1875] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:46.152498 31769 ts_tablet_manager.cc:1888] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:18:46.152602 31769 log.cc:1196] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/cdc5670f722b437198eaf65666e35377
I20240501 08:18:46.153297 29956 catalog_manager.cc:4830] TS 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593): tablet cdc5670f722b437198eaf65666e35377 (table rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]) successfully deleted
I20240501 08:18:46.449884 31765 raft_consensus.cc:1050] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de: attempting to promote NON_VOTER 309c676e56d04218a47e557e6a834030 to VOTER
I20240501 08:18:46.450541 31765 consensus_queue.cc:237] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:46.452839 31043 raft_consensus.cc:1261] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [term 1 LEARNER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:46.453275 30373 raft_consensus.cc:1261] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:46.453478 31823 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:46.453850 30243 raft_consensus.cc:1261] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:18:46.453856 31765 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:46.454526 31821 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:18:46.457579 31823 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.458215 30373 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.458830 30243 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.464093 31043 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.464074 29964 catalog_manager.cc:5484] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: config changed from index 4 to 5, 309c676e56d04218a47e557e6a834030 (127.2.248.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:46.468770 30109 consensus_queue.cc:237] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } }
I20240501 08:18:46.470571 30373 raft_consensus.cc:1261] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:46.471308 31044 raft_consensus.cc:1261] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:18:46.471616 31765 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:46.472231 31821 consensus_queue.cc:1035] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:18:46.474401 31824 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.474963 31043 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.475167 30373 raft_consensus.cc:2932] T 57bad390faac4b7f91c810137555662e P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } }
I20240501 08:18:46.477110 29956 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57bad390faac4b7f91c810137555662e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:46.478560 29967 catalog_manager.cc:5484] T 57bad390faac4b7f91c810137555662e P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: config changed from index 5 to 6, VOTER 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130) evicted. New cstate: current_term: 1 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:46.482304 30223 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bad390faac4b7f91c810137555662e with delete_type TABLET_DATA_TOMBSTONED (TS 7ef43f26720f4f4bbba3ebd7be22c26a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50852
I20240501 08:18:46.482988 31828 tablet_replica.cc:330] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: stopping tablet replica
I20240501 08:18:46.483196 31828 raft_consensus.cc:2229] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:46.483564 31828 raft_consensus.cc:2258] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:46.484431 31828 ts_tablet_manager.cc:1875] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:46.488444 31828 ts_tablet_manager.cc:1888] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:18:46.488530 31828 log.cc:1196] T 57bad390faac4b7f91c810137555662e P 7ef43f26720f4f4bbba3ebd7be22c26a: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-1/wal/wals/57bad390faac4b7f91c810137555662e
I20240501 08:18:46.489145 29957 catalog_manager.cc:4830] TS 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021): tablet 57bad390faac4b7f91c810137555662e (table rebalance_test_table_2 [id=9c40939ddc9a413cb3eef026c0b1819e]) successfully deleted
I20240501 08:18:46.615862 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:46.624049 30091 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:46.646971 30353 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:46.649698 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:46.650143 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:46.799127 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:46.804509 30091 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:46.805189 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:46.815567 30353 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:18:46.818066 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:46.886399 31043 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: true } }
I20240501 08:18:46.888561 30109 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:18:46.888679 30243 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 08:18:46.888968 30979 consensus_peers.cc:485] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 -> Peer 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593): Couldn't send request to peer 7e7b5a0a22114297942fe102e2cbd890. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:46.889323 31720 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:46.889636 31910 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:46.891868 31720 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: true } } }
I20240501 08:18:46.892637 30243 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: true } } }
I20240501 08:18:46.898690 30109 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: true } } }
I20240501 08:18:46.902503 29964 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a reported cstate change: config changed from index 7 to 8, NON_VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) added. New cstate: current_term: 2 leader_uuid: "309c676e56d04218a47e557e6a834030" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: true } } }
I20240501 08:18:46.916000 30373 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: true } }
I20240501 08:18:46.918123 31043 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:18:46.918366 30243 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:18:46.918761 31919 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:18:46.919123 30309 consensus_peers.cc:485] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 -> Peer 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997): Couldn't send request to peer 23d1b1dde90249cf82ab29f1b44795de. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:46.919169 31846 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:18:46.921720 31541 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: true } } }
I20240501 08:18:46.922407 31043 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: true } } }
I20240501 08:18:46.924651 30243 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: true } } }
I20240501 08:18:46.925813 29964 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index 6 to 7, NON_VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) added. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:18:47.497014 31952 ts_tablet_manager.cc:902] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Initiating tablet copy from peer 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633)
I20240501 08:18:47.498467 31952 tablet_copy_client.cc:308] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.133:35633
I20240501 08:18:47.499320 31064 tablet_copy_service.cc:140] P 309c676e56d04218a47e557e6a834030: Received BeginTabletCopySession request for tablet cdc5670f722b437198eaf65666e35377 from peer 7e7b5a0a22114297942fe102e2cbd890 ({username='slave'} at 127.2.248.131:51281)
I20240501 08:18:47.499469 31064 tablet_copy_service.cc:161] P 309c676e56d04218a47e557e6a834030: Beginning new tablet copy session on tablet cdc5670f722b437198eaf65666e35377 from peer 7e7b5a0a22114297942fe102e2cbd890 at {username='slave'} at 127.2.248.131:51281: session id = 7e7b5a0a22114297942fe102e2cbd890-cdc5670f722b437198eaf65666e35377
I20240501 08:18:47.501760 31064 tablet_copy_source_session.cc:205] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:47.503108 31952 ts_tablet_manager.cc:1875] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:18:47.507405 31952 ts_tablet_manager.cc:1888] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 08:18:47.507634 31952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc5670f722b437198eaf65666e35377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:47.509821 31952 tablet_copy_client.cc:783] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:47.510053 31952 tablet_copy_client.cc:647] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:47.512058 31952 tablet_copy_client.cc:523] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:47.514530 31952 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap starting.
I20240501 08:18:47.516376 31957 ts_tablet_manager.cc:902] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Initiating tablet copy from peer 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593)
I20240501 08:18:47.517974 31957 tablet_copy_client.cc:308] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: tablet copy: Beginning tablet copy session from remote peer at address 127.2.248.131:34593
I20240501 08:18:47.518690 30394 tablet_copy_service.cc:140] P 7e7b5a0a22114297942fe102e2cbd890: Received BeginTabletCopySession request for tablet 80b5f6bc3ac842ff931e03ae3094df55 from peer 23d1b1dde90249cf82ab29f1b44795de ({username='slave'} at 127.2.248.129:51421)
I20240501 08:18:47.518865 30394 tablet_copy_service.cc:161] P 7e7b5a0a22114297942fe102e2cbd890: Beginning new tablet copy session on tablet 80b5f6bc3ac842ff931e03ae3094df55 from peer 23d1b1dde90249cf82ab29f1b44795de at {username='slave'} at 127.2.248.129:51421: session id = 23d1b1dde90249cf82ab29f1b44795de-80b5f6bc3ac842ff931e03ae3094df55
I20240501 08:18:47.520668 30394 tablet_copy_source_session.cc:205] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:18:47.521806 31957 ts_tablet_manager.cc:1875] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:18:47.525910 31957 ts_tablet_manager.cc:1888] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:18:47.526170 31957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b5f6bc3ac842ff931e03ae3094df55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:18:47.528359 31957 tablet_copy_client.cc:783] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: tablet copy: Starting download of 0 data blocks...
I20240501 08:18:47.528617 31957 tablet_copy_client.cc:647] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: tablet copy: Starting download of 1 WAL segments...
I20240501 08:18:47.530270 31957 tablet_copy_client.cc:523] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:18:47.532974 31957 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap starting.
I20240501 08:18:47.548465 31952 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:47.548846 31952 tablet_bootstrap.cc:492] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Bootstrap complete.
I20240501 08:18:47.549072 31952 ts_tablet_manager.cc:1367] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Time spent bootstrapping tablet: real 0.035s	user 0.036s	sys 0.000s
I20240501 08:18:47.549655 31952 raft_consensus.cc:348] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: true } }
I20240501 08:18:47.549919 31952 raft_consensus.cc:729] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Initialized, Role: LEARNER
I20240501 08:18:47.550113 31952 consensus_queue.cc:260] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: NON_VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: true } }
I20240501 08:18:47.550657 31952 ts_tablet_manager.cc:1398] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240501 08:18:47.551592 31064 tablet_copy_service.cc:342] P 309c676e56d04218a47e557e6a834030: Request end of tablet copy session 7e7b5a0a22114297942fe102e2cbd890-cdc5670f722b437198eaf65666e35377 received from {username='slave'} at 127.2.248.131:51281
I20240501 08:18:47.551767 31064 tablet_copy_service.cc:433] P 309c676e56d04218a47e557e6a834030: ending tablet copy session 7e7b5a0a22114297942fe102e2cbd890-cdc5670f722b437198eaf65666e35377 on tablet cdc5670f722b437198eaf65666e35377 with peer 7e7b5a0a22114297942fe102e2cbd890
I20240501 08:18:47.566613 31957 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:18:47.566996 31957 tablet_bootstrap.cc:492] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Bootstrap complete.
I20240501 08:18:47.567226 31957 ts_tablet_manager.cc:1367] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Time spent bootstrapping tablet: real 0.034s	user 0.030s	sys 0.002s
I20240501 08:18:47.567817 31957 raft_consensus.cc:348] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: true } }
I20240501 08:18:47.568022 31957 raft_consensus.cc:729] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Initialized, Role: LEARNER
I20240501 08:18:47.568253 31957 consensus_queue.cc:260] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: NON_VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: true } }
I20240501 08:18:47.568727 31957 ts_tablet_manager.cc:1398] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240501 08:18:47.569434 30394 tablet_copy_service.cc:342] P 7e7b5a0a22114297942fe102e2cbd890: Request end of tablet copy session 23d1b1dde90249cf82ab29f1b44795de-80b5f6bc3ac842ff931e03ae3094df55 received from {username='slave'} at 127.2.248.129:51421
I20240501 08:18:47.569545 30394 tablet_copy_service.cc:433] P 7e7b5a0a22114297942fe102e2cbd890: ending tablet copy session 23d1b1dde90249cf82ab29f1b44795de-80b5f6bc3ac842ff931e03ae3094df55 on tablet 80b5f6bc3ac842ff931e03ae3094df55 with peer 23d1b1dde90249cf82ab29f1b44795de
I20240501 08:18:47.945232 30373 raft_consensus.cc:1203] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240501 08:18:47.992878 30109 raft_consensus.cc:1203] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240501 08:18:48.460353 32008 raft_consensus.cc:1050] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030: attempting to promote NON_VOTER 7e7b5a0a22114297942fe102e2cbd890 to VOTER
I20240501 08:18:48.461040 32008 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } }
I20240501 08:18:48.463385 30373 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 LEARNER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:48.463457 30109 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:48.463521 30243 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:48.463974 32010 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:48.464314 32011 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:48.464671 32008 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:48.478073 32010 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.478601 30373 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.481333 30111 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.484578 29968 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index 8 to 9, 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "309c676e56d04218a47e557e6a834030" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.485275 30243 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { replace: true } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.497896 31043 consensus_queue.cc:237] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } }
I20240501 08:18:48.499713 30243 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:18:48.499747 30373 raft_consensus.cc:1261] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Refusing update from remote peer 309c676e56d04218a47e557e6a834030: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:18:48.500348 32010 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:18:48.500782 31985 consensus_queue.cc:1035] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:18:48.503232 32010 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.504344 30243 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.505836 30373 raft_consensus.cc:2932] T cdc5670f722b437198eaf65666e35377 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.509524 29965 catalog_manager.cc:5484] T cdc5670f722b437198eaf65666e35377 P 7ef43f26720f4f4bbba3ebd7be22c26a reported cstate change: config changed from index 9 to 10, VOTER 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) evicted. New cstate: current_term: 2 leader_uuid: "309c676e56d04218a47e557e6a834030" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { promote: false } } }
I20240501 08:18:48.511013 29957 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cdc5670f722b437198eaf65666e35377 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:48.513307 30091 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc5670f722b437198eaf65666e35377 with delete_type TABLET_DATA_TOMBSTONED (TS 23d1b1dde90249cf82ab29f1b44795de not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46596
I20240501 08:18:48.514104 32029 tablet_replica.cc:330] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: stopping tablet replica
I20240501 08:18:48.514297 32029 raft_consensus.cc:2229] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:48.514544 32029 raft_consensus.cc:2258] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:48.515596 32029 ts_tablet_manager.cc:1875] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:48.520509 32029 ts_tablet_manager.cc:1888] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 08:18:48.520642 32029 log.cc:1196] T cdc5670f722b437198eaf65666e35377 P 23d1b1dde90249cf82ab29f1b44795de: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-0/wal/wals/cdc5670f722b437198eaf65666e35377
I20240501 08:18:48.521530 29956 catalog_manager.cc:4830] TS 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997): tablet cdc5670f722b437198eaf65666e35377 (table rebalance_test_table_1 [id=e48d78432ae64fb9a524f8dc524526d6]) successfully deleted
I20240501 08:18:48.542001 31987 raft_consensus.cc:1050] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: attempting to promote NON_VOTER 23d1b1dde90249cf82ab29f1b44795de to VOTER
I20240501 08:18:48.542654 31987 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } }
I20240501 08:18:48.544520 30109 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 LEARNER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:18:48.544965 30243 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:18:48.545109 31043 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Refusing update from remote peer 7e7b5a0a22114297942fe102e2cbd890: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:18:48.545053 31987 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:48.545521 32035 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:48.545846 32038 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:18:48.548640 32035 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } } }
I20240501 08:18:48.549265 30109 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } } }
I20240501 08:18:48.549878 31043 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } } }
I20240501 08:18:48.553522 30243 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } } }
I20240501 08:18:48.554006 29967 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 reported cstate change: config changed from index 7 to 8, 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7e7b5a0a22114297942fe102e2cbd890" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:48.746441 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60168
I20240501 08:18:48.746641 30373 raft_consensus.cc:595] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:48.963017 30373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55"
dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60188
I20240501 08:18:48.963208 30373 raft_consensus.cc:595] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:18:49.113379 32063 raft_consensus.cc:979] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: : Instructing follower 23d1b1dde90249cf82ab29f1b44795de to start an election
I20240501 08:18:49.113590 31987 raft_consensus.cc:1067] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Signalling peer 23d1b1dde90249cf82ab29f1b44795de to start an election
I20240501 08:18:49.114320 30109 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55"
dest_uuid: "23d1b1dde90249cf82ab29f1b44795de"
 from {username='slave'} at 127.2.248.131:51551
I20240501 08:18:49.114522 30109 raft_consensus.cc:482] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:18:49.114621 30109 raft_consensus.cc:3037] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:49.116611 30109 raft_consensus.cc:504] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } }
I20240501 08:18:49.117287 30109 leader_election.cc:288] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 2 election: Requested vote from peers 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593), 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021), 309c676e56d04218a47e557e6a834030 (127.2.248.133:35633)
I20240501 08:18:49.117640 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7e7b5a0a22114297942fe102e2cbd890"
I20240501 08:18:49.117729 30243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a"
I20240501 08:18:49.117892 30373 raft_consensus.cc:3032] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:18:49.117851 31043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b5f6bc3ac842ff931e03ae3094df55" candidate_uuid: "23d1b1dde90249cf82ab29f1b44795de" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "309c676e56d04218a47e557e6a834030"
I20240501 08:18:49.117971 30373 raft_consensus.cc:729] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e7b5a0a22114297942fe102e2cbd890, State: Running, Role: LEADER
I20240501 08:18:49.117992 30243 raft_consensus.cc:3037] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:49.118078 31043 raft_consensus.cc:3037] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:49.118276 30373 consensus_queue.cc:260] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } }
I20240501 08:18:49.118685 30373 raft_consensus.cc:3037] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:18:49.119930 30243 raft_consensus.cc:2454] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 2.
I20240501 08:18:49.120721 31043 raft_consensus.cc:2454] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 2.
I20240501 08:18:49.121136 30047 leader_election.cc:302] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 23d1b1dde90249cf82ab29f1b44795de, 309c676e56d04218a47e557e6a834030, 7ef43f26720f4f4bbba3ebd7be22c26a; no voters: 
I20240501 08:18:49.121416 32052 raft_consensus.cc:2781] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:18:49.121425 30373 raft_consensus.cc:2454] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1b1dde90249cf82ab29f1b44795de in term 2.
I20240501 08:18:49.121546 32052 raft_consensus.cc:686] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 2 LEADER]: Becoming Leader. State: Replica: 23d1b1dde90249cf82ab29f1b44795de, State: Running, Role: LEADER
I20240501 08:18:49.121872 32052 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } }
I20240501 08:18:49.124258 29965 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: term changed from 1 to 2, leader changed from 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) to 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129). New cstate: current_term: 2 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:49.516884 30243 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:49.517367 32052 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:49.543421 31043 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:49.544087 32052 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:49.549108 30373 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:18:49.549629 32080 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7b5a0a22114297942fe102e2cbd890" member_type: VOTER last_known_addr { host: "127.2.248.131" port: 34593 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:18:49.552215 30109 consensus_queue.cc:237] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } }
I20240501 08:18:49.554059 31043 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:18:49.554123 30243 raft_consensus.cc:1261] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Refusing update from remote peer 23d1b1dde90249cf82ab29f1b44795de: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:18:49.555210 32080 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:18:49.555469 32052 consensus_queue.cc:1035] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [LEADER]: Connected to new peer: Peer: permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:18:49.557632 32080 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } } }
I20240501 08:18:49.558248 31043 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 309c676e56d04218a47e557e6a834030 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } } }
I20240501 08:18:49.558817 30243 raft_consensus.cc:2932] T 80b5f6bc3ac842ff931e03ae3094df55 P 7ef43f26720f4f4bbba3ebd7be22c26a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } } }
I20240501 08:18:49.561054 29956 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80b5f6bc3ac842ff931e03ae3094df55 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:18:49.561609 29967 catalog_manager.cc:5484] T 80b5f6bc3ac842ff931e03ae3094df55 P 23d1b1dde90249cf82ab29f1b44795de reported cstate change: config changed from index 8 to 10, VOTER 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131) evicted. New cstate: current_term: 2 leader_uuid: "23d1b1dde90249cf82ab29f1b44795de" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ef43f26720f4f4bbba3ebd7be22c26a" member_type: VOTER last_known_addr { host: "127.2.248.130" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309c676e56d04218a47e557e6a834030" member_type: VOTER last_known_addr { host: "127.2.248.133" port: 35633 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1b1dde90249cf82ab29f1b44795de" member_type: VOTER last_known_addr { host: "127.2.248.129" port: 34997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:18:49.565539 30353 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b5f6bc3ac842ff931e03ae3094df55 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7b5a0a22114297942fe102e2cbd890 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33992
I20240501 08:18:49.566123 32083 tablet_replica.cc:330] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: stopping tablet replica
I20240501 08:18:49.566290 32083 raft_consensus.cc:2229] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:18:49.566460 32083 raft_consensus.cc:2258] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:18:49.567180 32083 ts_tablet_manager.cc:1875] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:18:49.571822 32083 ts_tablet_manager.cc:1888] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 08:18:49.571985 32083 log.cc:1196] T 80b5f6bc3ac842ff931e03ae3094df55 P 7e7b5a0a22114297942fe102e2cbd890: Deleting WAL directory at /tmp/dist-test-task3vHYYh/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714551038306655-3042-0/raft_consensus-itest-cluster/ts-2/wal/wals/80b5f6bc3ac842ff931e03ae3094df55
I20240501 08:18:49.572769 29956 catalog_manager.cc:4830] TS 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593): tablet 80b5f6bc3ac842ff931e03ae3094df55 (table rebalance_test_table_0 [id=0036cb4bea5d452ea35aad86d66b9d0c]) successfully deleted
I20240501 08:18:49.676882 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:49.696827 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.702991 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:49.716351 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.717321 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.840317 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:49.845295 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:49.847276 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.851051 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.860618 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:50.438536 29964 ts_manager.cc:292] Set tserver state for 0492adfedf2f446dbfc904261c50475c to MAINTENANCE_MODE
I20240501 08:18:50.973908 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:50.976382 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:50.979424 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:50.994781 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:50.995453 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.158308 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.167796 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.168306 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:51.171005 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.171804 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.333469 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.353400 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:51.353410 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.363914 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.374212 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.509096 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.515870 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.516474 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:51.530694 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.537804 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.694238 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.694752 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:51.707404 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.729298 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.730331 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.882688 31155 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.888738 30091 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:18:51.889632 30353 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.906699 31024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.910611 30223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:52.080291  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 30029
I20240501 08:18:52.089215 30150 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:52.233659  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 30029
W20240501 08:18:52.259418 30310 connection.cc:535] server connection from 127.2.248.129:51421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:18:52.260007  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 30161
W20240501 08:18:52.260007 30309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.248.129:34997 (error 108)
W20240501 08:18:52.262387 30309 consensus_peers.cc:485] T 9c596e34a1384dcf897b53b01bd01447 P 7e7b5a0a22114297942fe102e2cbd890 -> Peer 23d1b1dde90249cf82ab29f1b44795de (127.2.248.129:34997): Couldn't send request to peer 23d1b1dde90249cf82ab29f1b44795de. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.129:34997: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:52.266327 30283 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:52.401399  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 30161
I20240501 08:18:52.427109  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 30291
I20240501 08:18:52.433683 30414 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:52.577302  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 30291
W20240501 08:18:52.604441 29955 connection.cc:535] server connection from 127.2.248.131:50725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:18:52.605140  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 31093
W20240501 08:18:52.607741 30979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34593: connect: Connection refused (error 111)
W20240501 08:18:52.609041 30979 consensus_peers.cc:485] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 -> Peer 7e7b5a0a22114297942fe102e2cbd890 (127.2.248.131:34593): Couldn't send request to peer 7e7b5a0a22114297942fe102e2cbd890. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.131:34593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:52.611784 31214 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:18:52.659364 30980 consensus_peers.cc:485] T cdc5670f722b437198eaf65666e35377 P 309c676e56d04218a47e557e6a834030 -> Peer 7ef43f26720f4f4bbba3ebd7be22c26a (127.2.248.130:46021): Couldn't send request to peer 7ef43f26720f4f4bbba3ebd7be22c26a. Status: Network error: Client connection negotiation failed: client connection to 127.2.248.130:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:18:52.716496  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 31093
I20240501 08:18:52.736145  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 30961
I20240501 08:18:52.741962 31084 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:52.868232  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 30961
I20240501 08:18:52.891405  3042 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3vHYYh/build/asan/bin/kudu pid 29939
I20240501 08:18:52.892525 30000 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:18:53.039330  3042 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3vHYYh/build/asan/bin/kudu with pid 29939
2024-05-01T08:18:53Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (22990 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (46022 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (494761 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>

 1 FAILED TEST